In the code you posted it seems that some variable names are inconsistent (curStyleName <-> currentStyleName, StyleList <-> stylesNames), but that might be fine depending on the context your code is in.
Anyway I also get the same error when running the function "Do_Edit3DStyles" from the API Example "Environment_Control".
When debugging it I saw, that it runs into the following assertion:
ODB assertion failed at P:\VMWD\Build\Sources\ObjectDatabase\ObjectDatabaseDll\Database\ODBDatabase.cpp 1275
assert id: A9CF0419-32EB-4DEB-A7EB-73B9F6B82C7F
description: can't open modification scope inside transaction (modification scope should be the outermost)
I then tried to wrap ACEnv_SetCurrent3DStyleID in ACAPI_CallCommand but that still gives the same error. I filled out a bit more detailed bug report for Graphisoft. In case I hear back from them I'll let you know.