changeset:pipe() not return all validation errors. Only return 1 validation error.

[okaprinarjaya]

Hi @williamthome ,

I have following code

-module(model_user).
-export([changeset/1, changeset/2]).

-define(TYPES, #{
    id => binary,
    username => binary,
    userpassword => binary,
    role => binary,
    created_at => tuple
}).
-define(PERMITTED_ALL_ATTR, maps:keys(?TYPES)).
-define(REQUIRED_ALL_ATTR, maps:keys(?TYPES)).

changeset(Params) ->
    changeset(#{}, Params).

changeset(Data, Params) ->
    Changeset = changeset:cast({Data, ?TYPES}, Params, ?PERMITTED_ALL_ATTR),
    changeset:pipe(Changeset, [
        changeset:validate_required(?REQUIRED_ALL_ATTR),
        changeset:validate_member(role, [<<"ADMIN">>, <<"MITRA">>])
    ]).

Then when I do / execute the validation: model_user:changeset(#{}).
I only get 1 validation error: [{id,{<<"is required">>,#{validation => is_required}}}]
My expectation I get 5 validation error in total.

But I only get the following:

{changeset,[],
           #{id => binary,role => binary,username => binary,
             userpassword => binary,created_at => tuple},
           [id,role,username,userpassword,created_at],
           #{},#{},
           [{id,{<<"is required">>,#{validation => is_required}}}],  %% I only get 1 validation error. How to get all validation errors?
           [undefined,<<>>]}

Thank you 🙏

[williamthome]

Hi @okaprinarjaya.

I'll carefully look at your issues and improve this lib for you.
It's been a long time since I've been doing it. I need to remember, improve, and document things.

I'll be back soon to answer your questions.
Thank you.

[okaprinarjaya]

Thank you very much @williamthome
I do really appreciate it.

[williamthome]

Hi @okaprinarjaya.
Sorry for the delayed response.

I've been doing some tests and found changeset very limited. It only validates maps and does not validate recursive values, which is bad for Erlang terms, especially for records. Changeset was inspired by the Ecto Changeset lib, so it fits Elixir terms well, but they differ from Erlang.

So I decided to create a new lib. It's only in my machine and not ready for use. I'll push it to a repo soon and will call it evan, a library to validate and normalize any Erlang term. I'm creating a new lib because its structure differs from changeset.

Your example would be something like this in evan:

-module(model_user).
-export([changeset/1, changeset/2]).

-include("evan.hrl").

changeset(Params) ->
    changeset(#{}, Params).

% Currently, there is no support for comparison of changes.
% I'm using macros to simplify the code, but they are only functions under the hood.
% Prefixed with 'v_*' validates values and with 'n_*' normalizes values.
changeset(_Data, Params) ->
    evan:cast(Params, [
        ?v_is_map,
        ?n_map_with_keys([id, username, userpassword, role, created_at]),
        ?v_required_map_keys([id, username, userpassword, role, created_at]),
        ?v_map_key(id, [?v_is_binary]),
        ?v_map_key(username, [?v_is_binary]),
        ?v_map_key(userpassword, [?v_is_binary]),
        ?v_map_key(role, [?v_is_member([<<"ADMIN">>, <<"MITRA">>])]),
        ?v_map_key(created_at, [?v_is_tuple])
    ]).

A simple EUnit test:

-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").

changeset_test() ->
    [
        ?assertNot(evan:is_valid(changeset(#{}, #{}))),
        ?assert(evan:is_valid(changeset(#{}, #{
            id => <<"123456">>,
            username => <<"foo">>,
            userpassword => <<"helloworld!">>,
            role => <<"ADMIN">>,
            created_at => {1970,01,01}
        }))),
        ?assertEqual([
            {id,[<<"not a binary">>]},
            {username,[<<"not a binary">>]},
            {userpassword,[<<"not a binary">>]},
            {role,[<<"not a member">>]},
            {created_at,[<<"not a tuple">>]}
        ], evan:unwrap_error_messages(changeset(#{}, #{
            id => not_a_binary,
            username => not_a_binary,
            userpassword => not_a_binary,
            role => not_a_role_member,
            created_at => not_a_tuple
        })))
    ].

-endif.

Testing it:

$ rebar3 eunit --module model_user
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling evan
===> Performing EUnit tests...
.
Finished in 0.020 seconds
1 tests, 0 failures

A really ugly example that I'm using to test evan:

-ifdef(TEST).
-include_lib("eunit/include/eunit.hrl").
-include("evan.hrl").

cast_map_(Map) ->
    cast(Map, [
        ?v_is_map,
        ?n_map_with_keys([foo, bar, baz]),
        ?v_required_map_keys([foo, bar, baz]),
        ?v_map_key(foo, [?v_is_binary, ?n_binary_to_atom]),
        ?v_map_key(bar, [?v_is_binary, ?n_binary_to_atom]),
        ?v_map_key(baz, [
            % 'ref' is also made via 'refs' in cast options:
            % > cast(Value, Validations, #{
            %     refs => #{
            %         baz => [
            %             ?v_is_map,
            %             ?n_map_with_keys([fizz, buzz]),
            %             ?v_required_map_keys([fizz]),
            %             ?v_map_key(fizz, [?v_is_list, ?v_list_elems([?v_is_binary])]),
            %             ?v_map_key(buzz, [?c_ref(baz)])
            %         ]
            %     }
            % })
            % Doing it via macros, it will only be available for children
            % and not globally.
            ?c_make_ref(baz, [
                ?v_is_map,
                ?n_map_with_keys([fizz, buzz]),
                ?v_required_map_keys([fizz]),
                ?v_map_key(fizz, [?v_is_list, ?v_list_elems([?v_is_binary])]),
                % Using a ref turns the buzz key recursive.
                ?v_map_key(buzz, [?c_ref(baz)])
            ])
        ])
    ]).

cast_test() ->
    [
        ?assertNot(is_valid(cast_map_(#{
            % missing required fields
        }))),
        ?assertNot(is_valid(cast_map_(#{
            foo => <<"foo">>,
            bar => <<"bar">>,
            baz => #{
                fizz => [],
                buzz => [] % must be a map
            }
        }))),
        ?assertNot(is_valid(cast_map_(#{
            foo => <<"foo">>,
            bar => <<"bar">>,
            baz => #{
                fizz => [],
                % recursive validation
                buzz => #{
                    fizz => [],
                    buzz => #{
                        % missing fizz key
                    }
                }
            }
        }))),
        ?assert(is_valid(cast_map_(#{
            foo => <<"foo">>,
            bar => <<"bar">>,
            baz => #{fizz => []}
        }))),
        ?assertMatch(#{
            foo := foo,
            bar := bar,
            baz := #{fizz := []}
        }, value_out(cast_map_(#{
            foo => <<"foo">>,
            bar => <<"bar">>,
            baz => #{fizz => []}
        })))
    ].

-endif.

I'll come with updates soon.
I appreciate your patience.

[okaprinarjaya]

Hi @williamthome ,

Thanks a lot for your effort, I do really appreciate it. I think there's no called "delayed" at all. Let's just explore it for fun and maybe a very little profit someday in the future hahaha.. *kidding :D

Ok I'll test the new library "evan" once it is released.

Oya, did you know about https://livr-spec.org/ ? I think it will be good for evan to use the standard specification.

[williamthome]

Oya, did you know about https://livr-spec.org/ ? I think it will be good for evan to use the standard specification.

Oh, nice! As I see, there are two libs with this specification in Erlang:

• https://github.com/erlangbureau/liver
• https://github.com/Prots/olifer

Maybe I've not looked enough, but they cannot validate lists and tuples, only maps and proplists as the "main/initial" term. Also, there is no "normalization". Traverse the value again is necessary to normalize values. These are things that Evan implements.. validate and normalize any Erlang term and recursive (if needed).

The design of Evan is not ready yet, things may change, but taking your example again:

changeset(_Data, Params) ->
    evan:cast(Params, [
        % ...
        ?v_map_key(userpassword, [
            % Ensure it is a binary
            ?v_is_binary,
            % If is a binary, hash it
            fun({Pwd, State}) ->
                Iterations = 1_000,
                Salt = <<"oZzuwZvsSHutZFBt7hmA4gcBH9rsByEY">>,
                Bits = 32,
                Hash = crypto:pbkdf2_hmac(sha256, Pwd, Salt, Iterations, Bits),
                {ok, {Hash, State}}
            end
        ])
        %...
    ]).

By calling it:

1> model_user:changeset(#{}, #{userpassword => <<"mystrongpwd">>}).
{state,undefined,
       #{userpassword =>
             <<83,69,163,76,96,110,72,52,231,236,116,214,235,254,126,
               120,116,175,92,232,200,152,231,213,91,...>>},
       #Fun<evan.0.95663729>,true,#{},[]}
2> model_user:changeset(#{}, #{userpassword => not_a_binary}).
{state,undefined,
       #{userpassword => not_a_binary},
       #Fun<evan.0.95663729>,true,#{},
       [{{map_key,userpassword},
         [{bad_binary,#{message => <<"not a binary">>,
                        module => evan_v_is_binary,value => not_a_binary,
                        state =>
                            {state,not_a_binary,undefined,
                                   #Fun<evan.0.95663729>,false,#{},[]}}}]}]}

I'll try to release Evan v0.1.0 this weekend. Currently, finding free time for these libraries is not easy for me.
Thanks for the LIVR link. I'll look into it better.

[williamthome]

I'm sorry, @okaprinarjaya. I've been working on a library called doctest, which is useful to me and is now in a useful stage. This week, I plan to release the first version of a library that could help you. Let me know if you have any new insights or ideas. I appreciate your patience 🙏

[okaprinarjaya]

Hi @williamthome ,
Please don't worry, please don't think or threat me as your client. Let's just explore it for fun . Because I know your contribution / work / vision on erlang-web is to make web development in erlang as fun as in elixir. And I think, I also have same vision. Also i will learn a lot from you.

[williamthome]

Okay, great! I am working on closing or reducing some gaps for Erlang web developers. I enjoy making an effort to contribute to that.

[williamthome]

Hi, @okaprinarjaya!

After your last comment, I started revisiting Arizona (and I'll continue doing this for a while). Arizona is a web framework for the Erlang community. Its goal is to be simple, performative, and fun to use, as you said. I won a prize with it in the 2023 SpawnFest.

If you are interested, please feel free to follow the Arizona organization and star the project. I hope that the Erlang community will appreciate the project because there is currently no "plug and play" web framework for the language. While it's currently still under development and not ready for immediate use, the progress being made is incredibly promising. My idea is to have "changeset" and similar tools in Arizona.

Thanks for your comments and support!

Best regards.