Showing posts with label scala. Show all posts
Showing posts with label scala. Show all posts

Monday, September 23, 2019

Design patterns - no more

Design patterns - choose good programming language without internal "philosophical" gaps, then you don't need to learn patterns!

Yes, I admit I have aversion to patterns, for a simple reason - if programming language that you choose is good ... you don't need patterns, you don't need workarounds to cover language issues.

In my opinion patterns are just workarounds/cover-ups for language gaps/inconsistency.
It looks like: you build the house on the tree and you forgot to put a ladder, so you cannot reach the doors. Then comes patterns: use ropes, build concrete staircases, etc...
If you spend time on planing tree-house in the first place - no need for workarounds.

Just to be clear yes I'm talking about java - if you want to write a good program you need to know the whole catalog of patterns (starting with GOF). If you compare it to python, scala (tobe checked: kotlin, swift) - no more patterns. There are some suggested good behaviors like use functional instead of OO, but that is your choice/ your style.

I think this is an advantage of new languages - if you want to promote a new language you need to have it clean and easy to learn and use.




Friday, April 24, 2015

Fast data processing with spark - review

Hi,
I've got that book "Fast data processing with spark" and yesterday I spend on reviewing it..
I will not go into deep details, just quick one: it was quite disappointing book:

  • If you interested in one language (aka scala) then 1/3 of book is useless, as author is providing every example in 3 languages: scala/java/python.
  • Together there are ... 2 examples, so not much to share/learn.
  • Second example uses library that you cannot find anymore :-(, I mean - maybe if you spent more time to dig particular problem on geolocation - but do not use that book as geolocation solution. For general publicity not worth.
  • Explanations what's going on on examples are really light - and this is me being polite. 

Book: Not recommended.

If you want to check "working solution" updated to 2015 code base with existing libraries, check my project on github: 



Wednesday, March 25, 2015

LWJGL3 - use GPU [OpenCL] for multithreaded calculations

Lastly I spent some time investigating usage of GPU in computations, particularly nvidia CUDA. But during research I spent some time looking for other options, like OpenCL. So that brings me to this Lightweight Java Game Library.

Short intro:

LWJGL is a Java library that enables cross-platform access to popular native APIs useful in the development of graphics (OpenGL), audio (OpenAL) and parallel computing (OpenCL) applications. This access is direct and high-performance, yet also wrapped in a type-safe and user-friendly layer, appropriate for the Java ecosystem. LWJGL is an enabling technology and provides low-level access. It is not a framework and does not provide higher-level utilities than what the native libraries expose. As such, novice programmers are encouraged to try one of the frameworks or game engines that make use of LWJGL, before working directly with the library.   

And after quick look - you can get the same amount of access to all part of GPU as you can on JCUDA ( using CUDA of course ;-) ).

Here is simple OpenCL example  updated to use LWJGL3:




package com.yarenty.lwjgl3;

import org.lwjgl.opencl.CLUtil;
import org.lwjgl.BufferUtils;
import org.lwjgl.PointerBuffer;

import java.nio.FloatBuffer;
import java.nio.IntBuffer;
import java.util.List;

import org.lwjgl.opencl.CL;
import org.lwjgl.opencl.CLCreateContextCallback;
import org.lwjgl.opencl.CLDevice;
import org.lwjgl.opencl.CLPlatform;

import static org.lwjgl.opencl.CL10.*;
import static org.lwjgl.opencl.CLUtil.checkCLError;
import static org.lwjgl.system.MemoryUtil.NULL;
import static org.lwjgl.system.MemoryUtil.memDecodeUTF8;
 
public class OpenCLSum {
    // The OpenCL kernel
    static final String source =
        "kernel void sum(global const float *a, global const float *b, global float *answer) { "
        + "  unsigned int xid = get_global_id(0); "
        + "  answer[xid] = a[xid] + b[xid];"
        + "}";
 
    // Data buffers to store the input and result data in
    static final FloatBuffer a = toFloatBuffer(new float[]{1, 2, 3, 4, 5, 6, 7, 8, 9, 10});
    static final FloatBuffer b = toFloatBuffer(new float[]{9, 8, 7, 6, 5, 4, 3, 2, 1, 0});
    static final FloatBuffer answer = BufferUtils.createFloatBuffer(a.capacity());
 
    
 private static final CLCreateContextCallback CREATE_CONTEXT_CALLBACK = new CLCreateContextCallback() {
  @Override
  public void invoke(long errinfo, long private_info, long cb, long user_data) {
   System.err.println("[LWJGL] cl_create_context_callback");
   System.err.println("\tInfo: " + memDecodeUTF8(errinfo));
  }
 };
    
    public static void main(String[] args) throws Exception {
     
     System.setProperty("org.lwjgl.opencl.explicitInit","true");
     
     
        // Initialize OpenCL and create a context and command queue
        CL.create();
        System.out.println("CL created");
        
        CLPlatform platform = CLPlatform.getPlatforms().get(0);
        System.out.println("Platform created");

        PointerBuffer ctxProps = BufferUtils.createPointerBuffer(3);
  ctxProps.put(CL_CONTEXT_PLATFORM).put(platform).put(0).flip();
  System.out.println("CTX created");
  
  
        IntBuffer errcode_ret = BufferUtils.createIntBuffer(1);
  System.out.println("ERRCODE created");

        
        List devices = platform.getDevices(CL_DEVICE_TYPE_GPU);
       // long context = clCreateContext(platform, devices, null, null, null);
  long context = clCreateContext(ctxProps, devices.get(0).getPointer(), CREATE_CONTEXT_CALLBACK, NULL, errcode_ret);
  System.out.println("CONTEXT created");

  
  checkCLError(errcode_ret);
        //CLCommandQueue queue = clCreateCommandQueue(context, devices.get(0), CL_QUEUE_PROFILING_ENABLE, null);
        long queue = clCreateCommandQueue(context, devices.get(0).getPointer(), CL_QUEUE_PROFILING_ENABLE, errcode_ret);
  System.out.println("Command Q created");

        // Allocate memory for our two input buffers and our result buffer
        long aMem = clCreateBuffer(context, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, a, null);
        //long buffer = clCreateBuffer(context, CL_MEM_READ_ONLY, 128, errcode_ret);
  System.out.println("A Buffer created");
        clEnqueueWriteBuffer(queue, aMem, 1, 0, a, null, null);
        long bMem = clCreateBuffer(context, CL_MEM_READ_ONLY | CL_MEM_COPY_HOST_PTR, b, null);
  System.out.println("B Buffer created");
        clEnqueueWriteBuffer(queue, bMem, 1, 0, b, null, null);
        long answerMem = clCreateBuffer(context, CL_MEM_WRITE_ONLY | CL_MEM_COPY_HOST_PTR, answer, null);
  System.out.println("OUTPUT Buffer created");
        clFinish(queue);
 
        // Create our program and kernel
        long program = clCreateProgramWithSource(context, source, null);
  System.out.println("PROGRAM created");

     //public static int clBuildProgram(long program, long device, CharSequence options, CLProgramCallback pfn_notify, long user_data) {

      
        CLUtil.checkCLError(clBuildProgram(program, devices.get(0).getPointer(), "", null, 0L));
        // sum has to match a kernel method name in the OpenCL source
        long kernel = clCreateKernel(program, "sum", null);
  System.out.println("KERNEL created");

        // Execution our kernel
        PointerBuffer kernel1DGlobalWorkSize = BufferUtils.createPointerBuffer(1);
        System.out.println("KERNEL work size created");
        kernel1DGlobalWorkSize.put(0, a.capacity());
        System.out.println("KERNEL work size copied");
        
        clSetKernelArg1p(kernel, 0, aMem);
        clSetKernelArg1p(kernel, 1, bMem);
        clSetKernelArg1p(kernel, 2, answerMem);
        
  System.out.println("Args send to kernel");

        clEnqueueNDRangeKernel(queue, kernel, 1, null, kernel1DGlobalWorkSize, null, null, null);
        System.out.println("KERNEL queued created");
        
        
        // Read the results memory back into our result buffer
        clEnqueueReadBuffer(queue, answerMem, 1, 0, answer, null, null);
        System.out.println("and output ... created");
        
        clFinish(queue);
        // Print the result memory
        print(a);
        System.out.println("+");
        print(b);
        System.out.println("=");
        print(answer);
 
        // Clean up OpenCL resources
        clReleaseKernel(kernel);
        clReleaseProgram(program);
        clReleaseMemObject(aMem);
        clReleaseMemObject(bMem);
        clReleaseMemObject(answerMem);
        clReleaseCommandQueue(queue);
        clReleaseContext(context);
        CL.destroy();
    }
 
 
    /** Utility method to convert float array to float buffer
     * @param floats - the float array to convert
     * @return a float buffer containing the input float array
     */
    static FloatBuffer toFloatBuffer(float[] floats) {
        FloatBuffer buf = BufferUtils.createFloatBuffer(floats.length).put(floats);
        buf.rewind();
        return buf;
    }
 
 
    /** Utility method to print a float buffer
     * @param buffer - the float buffer to print to System.out
     */
    static void print(FloatBuffer buffer) {
        for (int i = 0; i < buffer.capacity(); i++) {
            System.out.print(buffer.get(i)+" ");
        }
        System.out.println("");
    }
 
}

and you can find full scala project here: https://github.com/yarenty/scala_lwjgl3

Friday, March 20, 2015

Defaults - base start configurations for new projects.

Every single time I'm starting new projects there is time which I spent on "organising" stuff:
- for maven create initial pom.xml
- for scala  - build.sbt
- or define gitignore for different type of projects.
I know that there is some handy tools like gitignore generators, etc...
Tell you truth every single time I finish with going through all my previous projects and trying to find something close to the one that I'm doing at the moment. Then look into libraries to find if there are some new ones worth upgrading ...  Yes, i know it could be fun, but ...

Finally I decided to do some life hack and have my BASE DEFAULTS -  I created this project:  https://github.com/yarenty/DEFAULTS

And hope I will keep them updated.


PS: Apart standard one I'm keeping stuff like "must install in new eclipse"

Findbugs:
http://findbugs.cs.umd.edu/eclipse
PMD:
http://sourceforge.net/projects/pmd/files/pmd-eclipse/update-site/
Checkstyle:
http://eclipse-cs.sourceforge.net/update/
or new marketplace version:
http://eclipse-cs.sourceforge.net/

Wednesday, December 10, 2014

Scala1: Scala is craaazy

I finished scala course provided by Martin Odersky.  And just want to test them in some real case scenario. I had quite easy task: there is csv file, with headers, which need to be imported to database. Normal times when I used java I prepared myself for few days of coding. Typical steps are:

  1. csv: find/list java csv processing libraries
  2. csv: read / investigate few of them
  3. csv: start playing, when one do not work as expected - check another one
  4. db: fight with proper driver
  5. db: find how to connect
  6. db: prepare statements/sqls/ etc..
  7. finally connect csv+db into one solution, and make it work smoothly

And after few days I would have working solution with probably few hundred lines of code...(not to mention tests/documentation).

In scala (and I am novice):
  1. created my single object "importer.scala"
  2. whole csv solution finished with adding to my build.sbt file single line:

libraryDependencies += "com.github.tototoshi" %% "scala-csv" % "1.1.2"

as my csv file using ";" as separator I put :
  implicit object MyFormat extends DefaultCSVFormat {
    override val delimiter: Char = ';'  }

then to main function I added 2 lines of code:
    val reader = CSVReader.open("myfile.csv")(MyFormat)
    val full = reader.allWithHeaders

and ... that is it - whole csv solution done in less than 15 mins! (and yes in java I also used sbt like solution - i.e.:maven)

  3. DB part: 
Connection - again to build.sbt, added just one line (I'm using MySQL):
libraryDependencies += "mysql" % "mysql-connector-java" % "5.1.34"

..  and again to main function just added database connection
 val dbc = "jdbc:mysql://localhost:3306/mydb?user=me&password=123"
    Class.forName("com.mysql.jdbc.Driver")   
val conn = DriverManager.getConnection(dbc)

and coding ... lets go ;-) :
    try {
      full.foreach(fields => processFields(fields, conn))
    } finally {
      conn.close
    }


I know that I could do everything inside "full.foreach" but I had some old java habits to create methods for different actions... so I just added processFields function.

  def processFields(in: Map[String, String], conn: java.sql.Connection) {
    val l: MutableList[String] = new MutableList[String]()
    fieldTypes.foreach(f =>
      l += (in.get(f).get).replace("\"", "\\\""))
    val sql = "INSERT INTO temp_products (" + myTypes.mkString(",") +      ") VALUES ( " + "\"" + l.mkString("\",\"") + "\")"
    val prep = conn.prepareStatement(sql)
    prep.executeUpdate
  } 

As you see I had issue with ["] sign - which sometimes appeared inside csv fields.
This is plain/native sql - and whole solution takes me less than 2 hours.

Must say: I was little disappointed with ... so little effort to make things work.
And all I had is 60 lines of code - I really don't see reason to write comments/documentation as this is so obvious (if you know scala) that basically anything more will be overhead.

In next post I will show you about slick - proper DB/ORM mapping in scala and .. how easy is to do reverse engineering - generate objects from DB tables and do some crazy sqls without sql ;-)

Kowalski: The Rust-native Agentic AI Framework Evolves to v0.5.0 🦀

  TL;DR: Kowalski v0.5.0 brings deep refactoring, modular architecture, multi-agent orchestration, and robust docs across submodules. If yo...