Posted	Nick	Remark
#openstack-nova - 2018-05-29
16:29:18	mriedem	mgagne: yeah probably at least pike, although i'm not sure if any of the internal interfaces i'm using in that CLI have changed since then
16:30:09	mriedem	mgagne: could be possible to setup a small pike test environment with the caching scheduler, create some servers, resize one or two of them, and then run the heal_allocations CLI
16:30:36	mriedem	i can also test that out if/when i get some free time (first day back after the summit)
16:31:24	mriedem	gibi: want to skip the notifications meeting today? i'm still getting caught up from last week
16:31:32	mgagne	mriedem: I suppose master would be best? I can try to setup a test environment in our infra. It would however be a non-trivial amount of effort as I'm not familiar with devstack and would probably do it "our way".
16:32:22	mriedem	mgagne: sure whatever you're most comfortable with is best
16:32:57	mgagne	mriedem: alright, I will attempt to squeeze time for that.
16:33:12	mriedem	great, thanks
16:34:58	mgagne	mriedem: btw, regarding cellsv1 -> cellsv2, a light bulb lighten up at the summit. I was previously under the impression that you *had* to run cellsv2 in ocata and that cellsv1 was removing/non-compatible, making a migration impossible due to me requiring support for multi-cells.
16:36:07	mriedem	nope
16:36:14	mriedem	so you're 4.5% happier now right? :)
16:36:28	mgagne	99% ? =)
16:36:50	mriedem	mgagne: our cells v1 CI job is still cells v1 with a single cell1 for the nova db in the child cell and cell0
16:37:01	mriedem	it's not multiple child cells, but still cells v1
16:37:04	mgagne	all that time... ^^'
16:37:50	mriedem	cool, glad it's more clear now
16:42:05	mriedem	jmlowe: did you get my 2nd email about checking the allocation records for the compute nodes that are failing to live migrate?
16:42:51	jmlowe	I did, turns out my allocations are a mess
16:42:56	gibi	mriedem: besides what I wrote in the summary mail today on the ML I have nothing else for the meeting
16:43:01	mriedem	gibi: ok
16:43:06	jmlowe	I cleaned up the double allocations
16:43:20	jmlowe	then I found allocations on the wrong host
16:43:22	gibi	mriedem: so I will open the meeting to see if somebody new shows up but will close it in 5 minutes
16:43:34	mriedem	jmlowe: hmm, ok. still weird that the problem didn't show up with normal server create.
16:43:38	jmlowe	I really could use that new force create of allocations
16:44:05	mriedem	jmlowe: the force option mentioned in my heal_allocations patch?
16:44:23	jmlowe	I'm whipping up something to check nova location vs placement allocation location
16:44:43	jmlowe	yeah that, my current workaround is to delete then live migrate
16:45:47	mriedem	ok. wonder how everything got so messed up in the first place, but like i said in the email, there were bugs in the pike GA regarding properly cleaning up allocations when something failed - those have all been fixed in stable pike point releases, but you could have had cruft from the GA
16:46:27	jmlowe	I figure I've got no chance of things working as expected unless all of the placement allocations are correct
16:48:03	jmlowe	I did have a thought, we do lots of shelving and unshelving, any chance allocations are getting correctly updated during unshelve?
16:48:24	jaypipes	jmlowe: you mean shelve_offloading?
16:48:39	jmlowe	yes
16:48:43	jaypipes	jmlowe: because I don't believe shelving (without offloading) affects allocations. could be wrong though.
16:49:01	jaypipes	shelve is possibly one of the worst designed things in Nova... </grumbling>
16:49:05	mriedem	when we shelve offload, the allocations for the node on which the instance is running will be deleted
16:49:42	mriedem	https://github.com/openstack/nova/blob/stable/pike/nova/compute/manager.py#L4496
16:50:07	dansmith	he said "without offloading"
16:50:11	mriedem	when you unshelve, you go through the scheduler which will create allocations against the selected destination host
16:50:44	mriedem	where did he say 'without offloading'?
16:51:00	jmlowe	Really need it for our workflows, scientist does data analysis and runs for several weeks, shelves and goes off to write up results, submits paper and waits a couple of months to hear back from reviewers, unshelves and reruns based on reviewers comments
16:51:01	dansmith	<jaypipes>jmlowe: because I don't believe shelving (without offloading) affects allocations. could be wrong though.
16:51:02	mriedem	jay said that
16:51:11	mriedem	by default shelve offloads immediately
16:51:15	mriedem	and removes allocatoins
16:51:18	dansmith	yeah, I thought you were contradicting jaypipes' comment
16:51:33	mriedem	no, i thought jay was asking for clarification
16:51:34	dansmith	jaypipes: I learned something about the importance of shelve this summit by the way
16:51:41	mriedem	to which jmlowe said "yes" to shelve_offload
16:51:46	jmlowe	that default behavior is why I tend to drop the offloading when referring to it
16:51:51	dansmith	jaypipes: related to my apparent misunderstanding of how AWS works
16:51:58	mriedem	jmlowe: same here
16:52:15	mriedem	jmlowe: so shelve / unshelve should be ok wrt allocations
16:52:44	mriedem	https://review.openstack.org/#/c/494632/
16:52:57	mriedem	that was in the pike GA also
16:53:01	jmlowe	I'll start spot checking history once I get a list of all the errant allocations
16:53:52	cfriesen	dansmith: I'll bite...what did you learn about shelve?
16:54:14	dansmith	cfriesen: weren't you in that cyborg session?
16:54:51	cfriesen	dansmith: if I was I missed the important bit. :)
16:55:34	dansmith	cfriesen: that the AWS equivalent of our stop/start involves a reschedule and is actually a little more equivalent to our shelve(-offload) and unshelve in terms of freeing resources (and the associated billing)
16:56:19	cfriesen	dansmith: ah, okay. makes sense. I never could figure out the logic of why we have stop/suspend/pause when they all still consume resources.
16:57:03	dansmith	well, just lifecycle of a pet, but I had assumed we imported that from AWS for parity
16:57:58	jmlowe	I'd kill for the ability to offline migrate a suspended instance btw
16:58:27	cfriesen	jmlowe: should be technically doable.
16:58:55	jmlowe	cfriesen: right, just move the definition and the file where the state is written?
16:59:34	cfriesen	jmlowe: essentially. the resource tracking and wiring up the networking and storage and whatnot are the tricky bits
16:59:57	dansmith	yeah, a little more complicated than that
16:59:58	dansmith	I thought that cold migrate could do it though
17:00:31	cfriesen	dansmith: if not, I bet it wouldn't be horrendous to add it
17:00:38	jmlowe	haven't tried for a while
17:01:01	dansmith	or do you mean actually suspended (i.e. memory state saved) ?
17:01:44	cfriesen	dansmith: looks like we only cold-migrate "active" or "stopped" instances
17:02:00	dansmith	yeah, I assumed he meant suspended==stopped
17:02:15	dansmith	I'm not sure I understand the rationale for suspend in the cloud either
17:02:30	dansmith	maybe faster boot for slow-to-start applications or something, but..
17:02:39	yumapath	hi can i get some help on putting pci devices in passthrough mode
17:02:48	yumapath	https://docs.openstack.org/nova/pike/admin/pci-passthrough.html
17:02:52	cfriesen	dansmith: in order to manage disk space on compute nodes we ended up making "suspend" behave like "pause"
17:03:04	yumapath	i followed this link , but not able to get things working
17:03:26	yumapath	if anyone out there can help me that would be great
17:05:14	dansmith	cfriesen: for what gain?
17:05:21	dansmith	oh,
17:05:23	dansmith	suspend like pause
17:05:29	dansmith	so they don't take up extra space with the memory image?
17:12:31	jmlowe	hmm, so it would seem out of 946 instances two were in the wrong place, about two dozen don't have allocations
17:14:14	jmlowe	last instance action for the one in the wrong place was an errored live migration
17:15:06	openstackgerrit	Merged openstack/nova master: Adding NVMEoF for libvirt driver https://review.openstack.org/482640
17:20:34	cfriesen	dansmith: was away from keyboard. yeah, otherwise we'd need to reserve as much disk space as we have guest RAM, which might be an issue on small-disk compute nodes.
17:21:03	dansmith	yeah, makes sense. unfortunately it changes the behavior the api user expects
17:23:44	cfriesen	dansmith: agreed. though the practical difference between the two is minimal
17:24:28	jaypipes	jmlowe: sorry, had a phone call... ok, sorry, hope you are ready for a stream of opinions coming from me... here goes...
17:24:49	dansmith	cfriesen: well, if you're using pause for fast HA, the startup time may differ significantly, but also suspend should survive host reboots, power outages, etc
17:24:53	dansmith	cfriesen: but yes, similar
17:25:09	jaypipes	jmlowe: a) application state should *never* be saved in a VM disk image. it should always be saved only to a persistent volume that can be attached/detached as needed.
17:25:58	jmlowe	great in theory, now fix 40 years of domain specific scientific code
17:26:13	jaypipes	jmlowe: b) if you need persistent IP addresses, then those IP addresses should be managed outside of the scope of Nova (i.e. created in Neutron and attached to a Nova VM on boot). Then detached from the VM when no longer needed.
17:26:25	jaypipes	jmlowe: like I said, my opinion... :)
17:27:44	jmlowe	case in point, most domestic weather forecasting is done with WRF, there is exactly one guy in the world who still knows how it works, he was brought out of retirement at UCAR, there is a new model in the works but I don't think it's ready for primetime yet
17:27:52	jaypipes	jmlowe: c) you will end up spending more money and time working around coupled app state and configuration instead of investing the time in decoupling the app configuration and state and treating all your VMs as cattle instead of pets.
17:29:43	jaypipes	jmlowe: ok, I'm done with my opinion stream. :) I recognize there are real and unworkaroundble constraints that you and others suffer with. I'm just pointing out my experience and long-term strategy, nothing more.

Earlier   Later