ClassFiltermachanism alone is completely ineffective in preventing scripts from invoking arbitrary Java code. According to Oracle this is the intended behavior, so you probably want to avoid Nashorn for running untrusted scripts.
The October 2018 CPU finally also applies classpath restrictions for LDAP object loading.
Up to the April 2018 CPU (6u191, 7u181, 8u171) Java’s RMI endpoints allowed HTTP tunneling of requests. Failing to implement further restrictions on these requests it was possible to perform them as cross-origin requests from third-party websites. This makes it possible to exploit otherwise unreachable RMI endpoints.
Executive summary, so that you don’t have a heart attack before we get into the gritty details.
CVE-2018-2633 - fixed in the January 2018 CPU - allows remote code execution under two conditions:
com.sun.security.enableAIAcaIssuers==true- which is hopefully as uncommon as a google search suggests, or
- CRL checking/downloads are enabled (mostly
com.sun.security.enableCRLDP==true, but also possibly other configurations) and the attacker can forge a otherwise valid/trusted certificate with an invalid CRL distribution point URL.
CVE-2017-10116 - fixed in the July 2017 CPU - possibly allowed code execution through Java deserialization for an attacker in a MITM position.
All of these apply to all regular X.509 certificate validation using Java’s built-in implementation, i.e. TLS client, TLS server (if client certificates are used), JAR verification… but only under aforementioned conditions.