add nsfw praxis notes for Azure #164#20
Conversation
sanity check for the two candidate models before APIPod integration. 5 safe test images through both models: bhky/opennsfw2 scores under 0.02 on all 5, Falconsai/nsfw_image_detection labels all 5 as normal at 99.67%+. then 5 classical art nudes from Wikimedia (public domain) to check the models actually catch nudity: bhky flags 3/5, Falconsai flags 2/5 and distinguishes sculpture from explicit art. tmplink/nsfw_detector from the ticket returns 404 on huggingface, subbed Falconsai as a maintained modern equivalent.
There was a problem hiding this comment.
I think the whole thing is missing a little bit the point.
Art like this actually should not really be classified as adult content/violence/nsfw.
I understand when does and is okay that it get's classified but if you want a test, Id rather recommend to take real pornographic and especially borderline content. A woman in a bikini is most likely not NSFW but nude at the beach yes.
Pure violence, gore, weapons, nsfw. ist also part of this. We don't want to be too picky; finding a good threshold is important.
We will run the model on AI inputs and AI outptus so it must be cheap and fast.
This is the necessary first step for having public content on our website.
The falcon model looks good please check if we can use it (licence) wise. It might be too heavy though.
An option might be to have it like fast detector if confidence < 0.8 --> Falcon or something similar.
Have you had a look at shieldgemma? Looks promising
https://huggingface.co/google/shieldgemma-2-4b-it
https://huggingface.co/google/shieldgemma-2b
It looks like this one can not only detect nude,violence etc. but also works for text which is great!
We can define then a policy
When porting it, check-out the qwen-model repository and the new_registry dev of APIPod because the syntax of models changed a lot.
Like always instead of blind activism think what we actually want to do. Then do research.
Then take action.
2 participants
what i did
ran the two candidate NSFW models from ticket Azure #164 locally to check them before writing the APIPod service. NSFW = filter that flags nudity, gore, or other adult content before it leaves the platform. as a MaaS provider, we need this on outputs so people don't build with us and end up shipping something they can't use in their app.
bhky/opennsfw2(yahoo openNSFW2, PyPI package, TF/Keras, single float [0,1]) plusFalconsai/nsfw_image_detection(ViT, transformers pipeline, {normal, nsfw} softmax) as a live substitute fortmplink/nsfw_detectorwhich returns 404 on huggingface today.5 synthetic safe images (plain colour, checkerboard, gradient, text, landscape) at 224x224. checked in under
docs/example_images/nsfw_praxis/safe/.results, safe pass
bhky/opennsfw2 scores the safe images from 0.0000 to 0.0119. threshold is 0.5, so well below. cold start 2.1s, warm ~265ms.
Falconsai/nsfw_image_detection labels all 5 as
normalat 99.67% or higher. cold start 2.1s, warm ~198ms.results, borderline
safe images alone only prove the models don't false-alarm on colours and shapes. they don't prove the models catch nudity. so a second pass with 5 classical art nudes from Wikimedia Commons (public domain): Botticelli Venus, Michelangelo David, Titian Venus of Urbino, Rubens Three Graces, Doryphoros. under
docs/example_images/nsfw_praxis/borderline/.bhky flags 3 of 5 (David, Titian, Rubens). misses Botticelli and Doryphoros. aggressive older CNN, more false positives on art.
Falconsai flags 2 of 5 (Titian, Rubens). reads the sculptures and softer Botticelli as normal. more nuanced modern ViT.
both models detect nudity, they don't just always answer safe. Falconsai looks like the better default for MaaS filtering, bhky useful as a stricter mode.
frictions worth flagging
tmplink/nsfw_detectorin the ticket no longer resolves on huggingface. subbed Falconsai. ticket may need an update or we go with the substitute.opennsfw2downloads weights on first predict. worker needs a writable cache dir or every cold start re-downloads.next
measure gpu latencies once the APIPod worker is up, throughput under load, and confirm the model choice with owner. APIPod integration itself is a separate PR.