| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-22 | |||
| 14:51:33 | mdbooth | And unfortunately I'm not always wrong. | |
| 14:53:40 | bnemec | It seems highly unlikely that a concurrency class would be non-reentrant. | |
| 14:53:57 | mdbooth | bnemec: s/non-// | |
| 14:54:17 | mdbooth | bnemec: I would *not* expect Semaphore to be re-entrant, and certainly in a regular environment it isn't: | |
| 14:54:35 | bnemec | Are you sure the tracing is in the right place? We spent a bunch of time tracking down a double-locking problem at one point that was caused by logging in the wrong place. | |
| 14:54:48 | mdbooth | >>> import threading >>> s = threading.Semaphore() | |
| 14:54:49 | mdbooth | >>> with s: | |
| 14:54:49 | mdbooth | ... with s: ... print("Hello world") | |
| 14:54:49 | mdbooth | ... | |
| 14:54:51 | mdbooth | Eurgh | |
| 14:54:52 | mdbooth | Sorry | |
| 14:55:02 | mdbooth | Anyway, takes same semaphore twice, hangs | |
| 14:55:40 | bnemec | Oh, well no that wouldn't work. But that's not what reentrancy is. | |
| 14:55:51 | mdbooth | bnemec: Wouldn't rule it out, but I don't think so. It's in a functional test, though, so my prime suspect is environmental. | |
| 14:55:53 | bnemec | Reentrancy has to do with concurrent execution, which you can't get in a single thread. | |
| 14:56:06 | mdbooth | bnemec: You might want to look that up ;) | |
| 14:56:17 | bnemec | I did. Did you? https://en.m.wikipedia.org/wiki/Reentrancy_%28computing%29 | |
| 14:56:25 | bnemec | "In computing, a computer program or subroutine is called reentrant if multiple invocations can safely run concurrently." | |
| 14:56:39 | mdbooth | https://docs.python.org/3/library/threading.html#semaphore-objects | |
| 14:57:07 | mdbooth | https://en.wikipedia.org/wiki/Reentrant_mutex | |
| 14:57:52 | bnemec | Ah, we're talking about different things. | |
| 14:59:35 | mdbooth | So, Semaphore doesn't appear to be mocked, and yet I see 2 enters followed by 2 exits | |
| 15:00:36 | openstackgerrit | Eric Fried proposed openstack/nova stable/train: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/690099 | |
| 15:00:36 | openstack | bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried) | |
| 15:01:20 | mdbooth | bnemec: To be clear, I'm *not* expecting to find a bug in oslo.concurrency. I'm expecting to find a fixture somewhere doing something I haven't thought of yet. | |
| 15:03:17 | mriedem | smells like a bro down in here https://www.youtube.com/watch?v=CHnbC1N_G6w | |
| 15:05:49 | mdbooth | Oh, great, it's a heisenbug >:( | |
| 15:07:39 | adrianc | sean-k-mooney, i can do the backport for https://review.opendev.org/#/c/675776/, however it will take me some time to get a setup up to test it | |
| 15:08:05 | bnemec | mdbooth: By any chance are you running tests in parallel? The external lock fixture from oslo.concurrency would make the locks independent. | |
| 15:09:36 | openstackgerrit | Dan Smith proposed openstack/nova master: Add image precaching docs for aggregates https://review.opendev.org/687348 | |
| 15:11:17 | openstackgerrit | Eric Fried proposed openstack/nova stable/train: Don't populate resources for not-yet-migrated inst https://review.opendev.org/690100 | |
| 15:11:39 | efried | there must be an easier way to cherry pick a series | |
| 15:15:23 | sean-k-mooney | efried: cerry picking a serise is baically a rebase | |
| 15:15:31 | sean-k-mooney | do you mean for backporting? | |
| 15:15:50 | efried | yes | |
| 15:15:55 | sean-k-mooney | ah | |
| 15:16:04 | sean-k-mooney | i was refering to a feature branch before | |
| 15:16:50 | sean-k-mooney | there is proably a way to do it in like 1 comamnd in git but i dont know what it is | |
| 15:17:48 | sean-k-mooney | oh git cherry-pick takes a list of commit | |
| 15:18:46 | sean-k-mooney | so if you do a git log betwen the base of the serise and top and just get the commits you shuld be able to pipe the revers of that to reverse and xargs to invoke the cherry pick | |
| 15:21:06 | efried | will that pile them up in a series? | |
| 15:21:15 | efried | that's probably the ticket, then. I'll try that next time. | |
| 15:21:29 | sean-k-mooney | i think so also looking at the exmaple in the git man pages | |
| 15:21:36 | sean-k-mooney | you might be able to do somting like this | |
| 15:21:46 | sean-k-mooney | git cherry-pick start..end | |
| 15:21:57 | dansmith | can't you do a range? | |
| 15:21:57 | sean-k-mooney | where start and end are two comites | |
| 15:21:58 | dansmith | yeah that | |
| 15:22:03 | dansmith | works for any other git command that takes multiple commits | |
| 15:22:31 | dansmith | if you have them all in a branch, then something like "git cherry-pick -x branch~4.." probably works | |
| 15:22:31 | sean-k-mooney | they have an example fo "git cherry-pick ..master" | |
| 15:22:48 | sean-k-mooney | that cherrypick anything on master but not locally | |
| 15:22:54 | sean-k-mooney | so i assuem you can close the range | |
| 15:23:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Require Migration object arg to migrate_instance_finish method https://review.opendev.org/690106 | |
| 15:24:49 | umbSublime | I have a specific workload where I need to enable invTSC, for this to work with migrations I also need to set the tsc frequency. How can I configure the tsc frequency in nova.conf on my compute hosts (is it possible with cpu_model_extra_flags)? in qemu it looks like this: "<clock><timer name='tsc' frequency='3000000000'/></clock>" | |
| 15:25:25 | sean-k-mooney | umbSublime: im pretty sure you cant | |
| 15:26:02 | sean-k-mooney | you could enabel a specific cpu flag but i dont think you can spcify the frequency | |
| 15:29:06 | umbSublime | Would the solution in this case be to create a custom CPU model and use that in nova.conf ? | |
| 15:29:16 | umbSublime | or is there an alternative workaround I didn't think about | |
| 15:29:54 | sean-k-mooney | well this is really a feature request so i would start with the uscase | |
| 15:30:33 | sean-k-mooney | you are trying to fake a specifc cpu frequece and provide a "stable" tsc to the guest correct? | |
| 15:30:39 | umbSublime | I'm only the operator for this cloud. But the use case is a software that requires that flag. Only issue is with only the flag enbaled and the frequency unset, migrations are not possible | |
| 15:30:43 | umbSublime | due to the nature of the flag | |
| 15:30:47 | sean-k-mooney | so that if you migrate betwwen hosts with differen cpu fequecne it does not chagne | |
| 15:31:10 | umbSublime | exactly there are some extensive discussions about that and how it was implemented in qemu/libvirt | |
| 15:31:51 | umbSublime | when you set a specific frequency, there is a calculation made so that migration is possible and the value of the register is mirrored over | |
| 15:32:44 | sean-k-mooney | ya so nova dose not currently suport hat | |
| 15:32:48 | sean-k-mooney | *that | |
| 15:32:51 | umbSublime | All I know is the software they run on the guest requires that invTSC is presented to the guest. From my operation pov I also need the VM to be migratable | |
| 15:33:00 | sean-k-mooney | we could ad support if we had a defied usecase | |
| 15:33:28 | sean-k-mooney | do they need invTSC set or do the want the TSC to be marked as reliable in the vm | |
| 15:33:31 | umbSublime | It's a game engine software so using other counters isn't really possible | |
| 15:33:41 | umbSublime | they need invTSC to be set | |
| 15:34:06 | sean-k-mooney | ok so setting tsc=reliable on the kernel commandline wont be viable | |
| 15:35:01 | sean-k-mooney | im basicaly wonder if https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/kernel-parameters.txt#L4832-L4838 | |
| 15:35:06 | umbSublime | Would the custom cpu_model be a working temporary solution ? | |
| 15:35:07 | sean-k-mooney | was another alternitive | |
| 15:35:14 | umbSublime | Ohh I didn't mention it's a windows guest :/ | |
| 15:35:20 | sean-k-mooney | oh :) | |
| 15:35:31 | sean-k-mooney | so that would be a know to linux kernel args then | |
| 15:35:48 | sean-k-mooney | in which case we shoudl be settin ghte hyperv features | |
| 15:37:14 | sean-k-mooney | if you modified the libvirt xml defition and created a new cpu model it might work | |
| 15:37:20 | sean-k-mooney | but i doubt it | |
| 15:37:45 | umbSublime | If I virsh edit a running vm and add the frequency, then reboot the VM it is migrated just fine by openstack | |
| 15:38:16 | sean-k-mooney | umbSublime: if you virsh edit it then it will be lost if nova regenerate the xml | |
| 15:38:24 | sean-k-mooney | e.g. via hard reboot or any move opertaiton | |
| 15:38:26 | umbSublime | Yes of course ~! | |
| 15:38:46 | sean-k-mooney | e.g. i would not expect it to survie a migraton | |
| 15:38:51 | sean-k-mooney | oh | |
| 15:38:54 | sean-k-mooney | you mean | |
| 15:38:55 | umbSublime | It was just to confirm that migration would work if forcing the freq. The next step was to configure in nova :) | |
| 15:39:10 | sean-k-mooney | before you live migrate you remove the invTSC flag | |
| 15:39:26 | umbSublime | nope both the invTSC and freq migrated with the VM | |
| 15:39:39 | umbSublime | I was actually surprised by that | |
| 15:40:02 | sean-k-mooney | well we dont update all the xml on live migration | |
| 15:40:05 | sean-k-mooney | we just do a diff | |
| 15:40:12 | sean-k-mooney | and modify what needs to be chagned | |
| 15:40:23 | umbSublime | Ok that explains it ! | |
| 15:40:26 | sean-k-mooney | but a hard reboot of the vm would loose any manual change you make | |
| 15:40:32 | umbSublime | yes absolutely | |