C++ is your best bet for a language. It is object-oriented and faster than Java. C# is an option but generally limited to Microsoft platforms.
Are you experienced in C++? Programming a game like VVVVV is not really 'simple'. Programming pong is simple.
Anyway... you will need to install OpenGL (http://en.wikipedia.org/wiki/OpenGL) for the graphics rendering. There are some complementary libraries you will also want to install (see Associated libraries section of wiki page above). The primary one you will want it either GLFW or freeglut.
When you are compiling the code you will want to put the OpenGL headers in the search path (e.g. -I/opt/X11/include) and link the OpenGL libraries (e.g. -l/opt/X11/lib/libGL.dylib -l/opt/X11/lib/libGLU.dylib in g++ main.cpp -o main -I/opt/X11/include -l/opt/X11/lib/libGL.dylib -l/opt/X11/lib/libGLU.dylib
Note, the paths above are SYSTEM-SPECIFIC depending on where the headers and libraries are installed (I am using Mac OS X Yosemite and macports as my package manager for GLFW/freeglut). I would recommend using CMake to generate the build rules if you plan to ever build the code on another machine because CMake has a find_package() routine that will take care of finding the headers and libraries on that system. Also with regards to the header search paths, this is assuming you are using the form #include 'GL/gl.h' for including the OpenGL headers in your source code, since gl.h is in /opt/X11/include/GL/gl.h.
Personally, I haven't done graphics rendering on Windows so the following is for OS X/Linux: when building, you also need to add the X11 folder to the header search path and link against the X11 libraries since X11 is the windowing system OS X/Linux runs OpenGL in.
I would start by making a simple test.cc file that has only a box or a sphere and trying to get it display before you do any coding specific for the game. Getting something as simple as this to compile and display properly when you are starting out can take quite a while when you first start out.
Once you can do this, my personal recommendation is that you will want to take advantage of the object-oriented and polymorphism of C++ to display your scene. For example, you would want to create an abstract base class (named maybe DrawableObject and has a pure virtual function 'draw()') that your game character and all the objects (obstacles, etc.) in the game inherit from and implement the 'draw()' function. This way, you can simply keep two containers: all the objects and a list of the objects in the current scene. When your calculations of which objects are currently in the scene (removing the objects that aren't in the scene from the aforementioned scene-only container), the location of the characters, etc. are finished, you just simply loop over the scene-only container of DrawableObjects and call the draw() function -- Side note, I'm recommending to do the calculations completely separate from the drawing/rendering. In other words, calculate everything and THEN draw everything; don't calculate one object, then draw that object, then calculate another, and so on... Imagine the graphics representation as built on top of the calculations such that you could turn off the visualization and the code would still run. This approach has the benefit of being more flexible (e.g. calculating the step of a character and not drawing would allow you to implement higher-order integration techniques such as Runge-Kutta 4th), the executable will likely run faster since it has to jump less to the rendering instructions in the linked libraries, and it will be much cleaner and easier to debug with the drawing routines separate from the calculation routines.