multimodal-learning

Here are 66 public repositories matching this topic...

pliang279 / awesome-multimodal-ml

Star 2.3k

Reading list for research topics in multimodal machine learning

machine-learning natural-language-processing reinforcement-learning computer-vision deep-learning robotics healthcare reading-list representation-learning speech-processing multimodal-learning

• Updated Jun 23, 2021

Eurus-Holmes / Awesome-Multimodal-Research

Star 666

A curated list of Multimodal Related Research.

awesome multimodal-learning multimodal multimodal-research

• Updated Jul 29, 2021
• Python

HuaizhengZhang / Awsome-Deep-Learning-for-Video-Analysis

Star 495

Papers, code and datasets about deep learning and multi-modal learning for video analysis

machine-learning deep-learning paper video-classification video-analysis multimodal-learning video-dataset

• Updated May 7, 2021

subho406 / OmniNet

Star 465

Official Pytorch implementation of "OmniNet: A unified architecture for multi-modal multi-task learning" | Authors: Subhojeet Pramanik, Priyanka Agrawal, Aman Hussain

nlp machine-learning deep-learning neural-network artificial-intelligence transformer image-captioning video-recognition multimodal-learning multitask-learning

• Updated Oct 31, 2020
• Python

PreferredAI / cornac

Star 389

A Comparative Framework for Multimodal Recommender Systems

collaborative-filtering matrix-factorization recommendation-system recommendation-engine recommender-system recommendation-algorithms multimodal-learning multimodal-representation

• Updated Jul 19, 2021
• Python

pykale / pykale

Star 253

Knowledge-Aware machine LEarning (KALE) from multiple sources in Python

machine-learning computer-vision pytorch transfer-learning graph-analysis domain-adaptation medical-image-analysis multimodal-learning knowledge-aware-learning

• Updated Jul 29, 2021
• Python

georgian-io / Multimodal-Toolkit

Star 146

Multimodal model for text and tabular data with HuggingFace transformers as building block for text data

natural-language-processing tabular-data transformer multimodal-learning huggingface-transformers

• Updated Jul 27, 2021
• Python

ArrowLuo / CLIP4Clip

Star 134

An official implementation for "CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval"

search retrieval ranking clip multimodality multimodal-learning multimodal activitynet retrieval-model msvd msrvtt video-text-retrieval lsmdc didemo video-clip-retrieval

• Updated Jul 26, 2021
• Python

machine-intelligence-laboratory / TopicNet

Star 108

Interface for easier topic modelling.

pypi topic-modeling multimodal-learning modalities document-representation multimodal-data topic-modelling bigartm-library custom-score

• Updated Jul 6, 2021
• Python

mhw32 / multimodal-vae-public

Star 87

A PyTorch implementation of "Multimodal Generative Models for Scalable Weakly-Supervised Learning" (https://arxiv.org/abs/1802.05335)

machine-learning variational-autoencoder generative-models multimodal-learning

• Updated Aug 17, 2018
• Python

sangminwoo / awesome-vision-and-language

Star 82

A curated list of awesome vision and language resources (still under construction... stay tuned!)

awesome awesome-list multimodal-learning vision-and-language

• Updated Mar 25, 2021

ABadCandy / BaiDuBigData19-URFC

Star 71

my solution with 0.67 accuracy

computer-vision deep-learning pytorch multimodal-learning

• Updated May 21, 2019
• Python

pliang279 / MFN

Star 69

[AAAI 2018] Memory Fusion Network for Multi-view Sequential Learning

machine-learning multimodal-learning

• Updated Aug 4, 2020
• Python

haamoon / mmtm

Star 53

Implementation of CVPR 2020 paper "MMTM: Multimodal Transfer Module for CNN Fusion"

pytorch action-recognition gesture-recognition multimodal-learning speech-enhancement multimodal-deep-learning cnn-fusion

• Updated Jun 16, 2020
• Python

PreferredAI / vista-net

Star 51

Code of the paper "VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis", AAAI'19

sentiment-analysis attention-mechanism multimodal-learning multimodal-sentiment-analysis

• Updated Sep 9, 2020
• Python

verlab / Learning2Dance_CAG_2020

Star 46

PyTorch implementation of our graph convolutional network (GCN) for human motion generation from music. Also with paired dance-music data for training!

computer-vision sound-processing graph-convolutional-networks gcn multimodal-learning motion-analysis motion-animation motion-synthesis human-motion human-motion-analysis graph-adversarial-learning computer-and-graphics

• Updated Apr 22, 2021
• Python

pliang279 / factorized

Star 45

[ICLR 2019] Learning Factorized Multimodal Representations

machine-learning representation-learning multimodal-learning

• Updated Aug 4, 2020
• Python

johnarevalo / gmu-mmimdb

Star 43

Source code for training Gated Multimodal Units on MM-IMDb dataset

representation-learning multimodal-learning

• Updated Nov 20, 2020
• Python

pliang279 / MultiBench

Star 38

Multiscale Benchmarks for Multimodal Representation Learning

machine-learning natural-language-processing computer-vision deep-learning robotics healthcare representation-learning speech-processing multimodal-learning

• Updated Jul 31, 2021
• Python

oswaldoludwig / visually-informed-embedding-of-word-VIEW-

Star 30

Visually informed embedding of word (VIEW) is a tool for transferring multimodal background knowledge to NLP algorithms.

nlp deep-learning word2vec embeddings mscoco multimodal-learning

• Updated Sep 18, 2016
• Python

anita-hu / MSAF

Star 24

Offical implementation of paper "MSAF: Multimodal Split Attention Fusion"

pytorch action-recognition multimodal-learning multimodal-sentiment-analysis multimodal-deep-learning ntu-rgbd cmu-mosei multimodal-emotion-recognition ravdess

• Updated Jun 16, 2021
• Python

antoyang / just-ask

Star 22

"Just Ask: Learning to Answer Questions from Millions of Narrated Videos", ICCV 2021 (oral)

vqa video-understanding weakly-supervised-learning multimodal-learning visual-question-answering question-generation vision-and-language videoqa pre-training video-question-answering

• Updated Jul 29, 2021
• Python

PreferredAI / mrg

Star 20

Code of the paper "Multimodal Review Generation for Recommender Systems", WWW'19

recommender-systems multimodal-learning review-generation multimodal-review-generation

• Updated Sep 19, 2020
• Python

kanchen-usc / VIG

Star 18

Dataset for Visually Indicated Sound Generation by Perceptually Optimized Classification

sound-synthesis video-analysis multimodal-learning

• Updated Apr 6, 2020
• Python

asnelt / mmae

Star 15

Package for Multimodal Autoencoders in TensorFlow / Keras

deep-learning tensorflow keras autoencoder autoencoders keras-models keras-tensorflow multimodal-learning multimodal-deep-learning bregman-distance

• Updated Jun 8, 2020
• Python

georgepar / slp

Star 15

Utils and modules for Speech Language and Multimodal processing using pytorch and pytorch lightning

natural-language-processing pytorch multimodal-learning multimodal multimodal-deep-learning pytorch-lightning wandb

• Updated Jul 6, 2021
• Python

Fly2flies / Cross-modal-retrieval

Star 14

媒体计算实践作业：图像——文本跨模态搜索

deeplearning multimodal-learning multimedia-retrieval

• Updated Dec 4, 2020
• Python

AnnikaLindh / Diverse_and_Specific_Image_Captioning

Star 14

Unsupervised specificity-guided optimization of Image Captioning models to encourage meaningful diversity in the generated captions.

machine-learning research computer-vision deep-learning pytorch image-captioning natural-language-generation paper-implementations multimodal-learning

• Updated Sep 17, 2019
• Python

ZihengZZH / bipolar-disorder

Star 12

Multimodal Deep Learning Framework for Mental Disorder Recognition @ FG'20

mental-health affective-computing multimodal-learning automatic-detection

• Updated May 21, 2021
• Python

wxl1999 / PLMPapers

Star 11

A paper list of pre-trained language models (PLMs).

multilingual machine-learning natural-language-processing deep-learning gpt representation-learning bert multimodal-learning pre-training

• Updated Jul 21, 2021

Load more…