| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-22 | |||
| 14:57:52 | bnemec | Ah, we're talking about different things. | |
| 14:59:35 | mdbooth | So, Semaphore doesn't appear to be mocked, and yet I see 2 enters followed by 2 exits | |
| 15:00:36 | openstackgerrit | Eric Fried proposed openstack/nova stable/train: Func: bug 1849165: mig race with _populate_assigned_resources https://review.opendev.org/690099 | |
| 15:00:36 | openstack | bug 1849165 in OpenStack Compute (nova) "_populate_assigned_resources raises "TypeError: argument of type 'NoneType' is not iterable" during active migration" [High,In progress] https://launchpad.net/bugs/1849165 - Assigned to Eric Fried (efried) | |
| 15:01:20 | mdbooth | bnemec: To be clear, I'm *not* expecting to find a bug in oslo.concurrency. I'm expecting to find a fixture somewhere doing something I haven't thought of yet. | |
| 15:03:17 | mriedem | smells like a bro down in here https://www.youtube.com/watch?v=CHnbC1N_G6w | |
| 15:05:49 | mdbooth | Oh, great, it's a heisenbug >:( | |
| 15:07:39 | adrianc | sean-k-mooney, i can do the backport for https://review.opendev.org/#/c/675776/, however it will take me some time to get a setup up to test it | |
| 15:08:05 | bnemec | mdbooth: By any chance are you running tests in parallel? The external lock fixture from oslo.concurrency would make the locks independent. | |
| 15:09:36 | openstackgerrit | Dan Smith proposed openstack/nova master: Add image precaching docs for aggregates https://review.opendev.org/687348 | |
| 15:11:17 | openstackgerrit | Eric Fried proposed openstack/nova stable/train: Don't populate resources for not-yet-migrated inst https://review.opendev.org/690100 | |
| 15:11:39 | efried | there must be an easier way to cherry pick a series | |
| 15:15:23 | sean-k-mooney | efried: cerry picking a serise is baically a rebase | |
| 15:15:31 | sean-k-mooney | do you mean for backporting? | |
| 15:15:50 | efried | yes | |
| 15:15:55 | sean-k-mooney | ah | |
| 15:16:04 | sean-k-mooney | i was refering to a feature branch before | |
| 15:16:50 | sean-k-mooney | there is proably a way to do it in like 1 comamnd in git but i dont know what it is | |
| 15:17:48 | sean-k-mooney | oh git cherry-pick takes a list of commit | |
| 15:18:46 | sean-k-mooney | so if you do a git log betwen the base of the serise and top and just get the commits you shuld be able to pipe the revers of that to reverse and xargs to invoke the cherry pick | |
| 15:21:06 | efried | will that pile them up in a series? | |
| 15:21:15 | efried | that's probably the ticket, then. I'll try that next time. | |
| 15:21:29 | sean-k-mooney | i think so also looking at the exmaple in the git man pages | |
| 15:21:36 | sean-k-mooney | you might be able to do somting like this | |
| 15:21:46 | sean-k-mooney | git cherry-pick start..end | |
| 15:21:57 | dansmith | can't you do a range? | |
| 15:21:57 | sean-k-mooney | where start and end are two comites | |
| 15:21:58 | dansmith | yeah that | |
| 15:22:03 | dansmith | works for any other git command that takes multiple commits | |
| 15:22:31 | dansmith | if you have them all in a branch, then something like "git cherry-pick -x branch~4.." probably works | |
| 15:22:31 | sean-k-mooney | they have an example fo "git cherry-pick ..master" | |
| 15:22:48 | sean-k-mooney | that cherrypick anything on master but not locally | |
| 15:22:54 | sean-k-mooney | so i assuem you can close the range | |
| 15:23:09 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Require Migration object arg to migrate_instance_finish method https://review.opendev.org/690106 | |
| 15:24:49 | umbSublime | I have a specific workload where I need to enable invTSC, for this to work with migrations I also need to set the tsc frequency. How can I configure the tsc frequency in nova.conf on my compute hosts (is it possible with cpu_model_extra_flags)? in qemu it looks like this: "<clock><timer name='tsc' frequency='3000000000'/></clock>" | |
| 15:25:25 | sean-k-mooney | umbSublime: im pretty sure you cant | |
| 15:26:02 | sean-k-mooney | you could enabel a specific cpu flag but i dont think you can spcify the frequency | |
| 15:29:06 | umbSublime | Would the solution in this case be to create a custom CPU model and use that in nova.conf ? | |
| 15:29:16 | umbSublime | or is there an alternative workaround I didn't think about | |
| 15:29:54 | sean-k-mooney | well this is really a feature request so i would start with the uscase | |
| 15:30:33 | sean-k-mooney | you are trying to fake a specifc cpu frequece and provide a "stable" tsc to the guest correct? | |
| 15:30:39 | umbSublime | I'm only the operator for this cloud. But the use case is a software that requires that flag. Only issue is with only the flag enbaled and the frequency unset, migrations are not possible | |
| 15:30:43 | umbSublime | due to the nature of the flag | |
| 15:30:47 | sean-k-mooney | so that if you migrate betwwen hosts with differen cpu fequecne it does not chagne | |
| 15:31:10 | umbSublime | exactly there are some extensive discussions about that and how it was implemented in qemu/libvirt | |
| 15:31:51 | umbSublime | when you set a specific frequency, there is a calculation made so that migration is possible and the value of the register is mirrored over | |
| 15:32:44 | sean-k-mooney | ya so nova dose not currently suport hat | |
| 15:32:48 | sean-k-mooney | *that | |
| 15:32:51 | umbSublime | All I know is the software they run on the guest requires that invTSC is presented to the guest. From my operation pov I also need the VM to be migratable | |
| 15:33:00 | sean-k-mooney | we could ad support if we had a defied usecase | |
| 15:33:28 | sean-k-mooney | do they need invTSC set or do the want the TSC to be marked as reliable in the vm | |
| 15:33:31 | umbSublime | It's a game engine software so using other counters isn't really possible | |
| 15:33:41 | umbSublime | they need invTSC to be set | |
| 15:34:06 | sean-k-mooney | ok so setting tsc=reliable on the kernel commandline wont be viable | |
| 15:35:01 | sean-k-mooney | im basicaly wonder if https://github.com/torvalds/linux/blob/master/Documentation/admin-guide/kernel-parameters.txt#L4832-L4838 | |
| 15:35:06 | umbSublime | Would the custom cpu_model be a working temporary solution ? | |
| 15:35:07 | sean-k-mooney | was another alternitive | |
| 15:35:14 | umbSublime | Ohh I didn't mention it's a windows guest :/ | |
| 15:35:20 | sean-k-mooney | oh :) | |
| 15:35:31 | sean-k-mooney | so that would be a know to linux kernel args then | |
| 15:35:48 | sean-k-mooney | in which case we shoudl be settin ghte hyperv features | |
| 15:37:14 | sean-k-mooney | if you modified the libvirt xml defition and created a new cpu model it might work | |
| 15:37:20 | sean-k-mooney | but i doubt it | |
| 15:37:45 | umbSublime | If I virsh edit a running vm and add the frequency, then reboot the VM it is migrated just fine by openstack | |
| 15:38:16 | sean-k-mooney | umbSublime: if you virsh edit it then it will be lost if nova regenerate the xml | |
| 15:38:24 | sean-k-mooney | e.g. via hard reboot or any move opertaiton | |
| 15:38:26 | umbSublime | Yes of course ~! | |
| 15:38:46 | sean-k-mooney | e.g. i would not expect it to survie a migraton | |
| 15:38:51 | sean-k-mooney | oh | |
| 15:38:54 | sean-k-mooney | you mean | |
| 15:38:55 | umbSublime | It was just to confirm that migration would work if forcing the freq. The next step was to configure in nova :) | |
| 15:39:10 | sean-k-mooney | before you live migrate you remove the invTSC flag | |
| 15:39:26 | umbSublime | nope both the invTSC and freq migrated with the VM | |
| 15:39:39 | umbSublime | I was actually surprised by that | |
| 15:40:02 | sean-k-mooney | well we dont update all the xml on live migration | |
| 15:40:05 | sean-k-mooney | we just do a diff | |
| 15:40:12 | sean-k-mooney | and modify what needs to be chagned | |
| 15:40:23 | umbSublime | Ok that explains it ! | |
| 15:40:26 | sean-k-mooney | but a hard reboot of the vm would loose any manual change you make | |
| 15:40:32 | umbSublime | yes absolutely | |
| 15:40:48 | sean-k-mooney | dumb question but do you neeed live migration. or would cold migration be enough | |
| 15:40:53 | umbSublime | this was just for testing purposes | |
| 15:41:23 | umbSublime | live-migration would really be the ideal goal (we need sometimes to update compute hosts without impacting guests) | |
| 15:42:06 | umbSublime | If this really isn't possible at the moment, I'm sure that aspect could be discussed internally | |
| 15:42:08 | sean-k-mooney | i think unfotuetly the best path forwad would be to add this as a new feature | |
| 15:42:15 | sean-k-mooney | what release of openstack are you running | |
| 15:43:13 | umbSublime | Currently that stack is on Queens | |
| 15:44:24 | sean-k-mooney | this is the only reference to invtsc in the nova code base | |
| 15:44:26 | sean-k-mooney | https://github.com/openstack/nova/blob/1ec87c227c4a506ddc3355108a95390cdaa57d62/nova/conf/libvirt.py#L608-L613 | |
| 15:45:32 | umbSublime | hehe yes, that's the only one I found also. Most of the work to make exposing that flag to the guest possible was made in qemu/libvirt projects. There many discussions in the MLs about this | |
| 15:46:21 | sean-k-mooney | umbSublime: can i suggest you file a bluepirnt for this. or maybe a bug fix. | |
| 15:47:25 | sean-k-mooney | i would like to call it a bug so it could be backproted but its really a feautre since invtsc frequesce configuration was never previously supproted so it shoudl be a specless blueprint not a bug | |
| 15:49:45 | umbSublime | sean-k-mooney: I'll add it to my todo list :) (I've never done any of those, but would gladly do it) | |
| 15:50:20 | umbSublime | Thanks a lot helping me out :) really appreciated | |
| 15:51:05 | sean-k-mooney | no worries. sorry i could not point you at an exisitng knob to tweak | |
| 15:51:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Stop converting Migration objects to dicts for migrate_instance_start https://review.opendev.org/690115 | |
| 15:59:49 | mriedem | dansmith: this is the requirements bump you'll need for your osc change https://review.opendev.org/#/c/690097/ | |
| 15:59:57 | mriedem | but the latest novaclient is breaking horizon... | |
| 15:59:59 | dansmith | mriedem: woot, I was just looking for it | |
| 16:00:05 | dansmith | guh | |