AbstractPhila PRO

AbstractPhil

https://civitai.com/user/AbstractPhila

AbstractEyes

AI & ML interests

datasets, research papers, experimentation, vision, classification, text encoders, tokenization, llms, diffusion, distillation, and more.

Recent Activity

replied to their post about 9 hours ago

I'll attempt to expand the geolip-clip to full sequence context window to encompass sequential learning. https://huggingface.co/AbstractPhil/geolip-clip-vit-large-patch14-ctx576 The memory pod is specifically meant to tune everything based on final state pooling, which is fine if you aren't trying to actually use sequential utility. HOWEVER, there are many elemental biases that present themselves if attempting to USE the standard sequence of 77 in conjunction with this final pooled state. Even though the standard 77 is predominantly noise past token 10 it still houses considerable amounts of information in terms of utility, so this should be handled carefully. Zero-shot structures are a tricky structure to analyze, especially structures based on attention mechanisms instead of true sequential accumulation. I've noticed I need to watch them for quite a while before the real bugs show up. As it stands the token pool is essentially [B, 7+8, 768] for pools. This contains a robust and highly complex representation of useful accumulated bidirectional attention data, so it's quite powerful. I'll build a few prototypes and tap into some papers. I'll either come up with something or a reason why I didn't. The end result will either produce an anchor bank set of tokens [B, 15, 768] for pooling, or [B, 15, 77, 768] ideally - which should expand the sequence of the clip to 1,155 if successful. That doesn't necessarily mean this sequence will be more useful than the [b, 15, 768], but it will be representationally valid to the context window expansion. I wouldn't hold out for a single full-sequence option in a single day, that's a lot of moving parts to analyze, not to mention highly impractical to train with. A smaller dose of this information would be necessary for rapid prototyping so it'll likely be packaged as such. Well I spoke too soon. It's ready to play with. https://huggingface.co/AbstractPhil/geolip-clip-vit-large-patch14-ctx576-seq77

updated a collection about 10 hours ago

GeoLIP

published a model about 10 hours ago

AbstractPhil/geolip-clip-vit-bigG-patch14-ctx576-seq77

View all activity

Organizations

AbstractPhil 's models 134

AbstractPhila PRO

AI & ML interests

Recent Activity

Organizations

AbstractPhil 's models 134 Sort: Recently updated

AbstractPhil 's models 134