| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-17 | |||
| 15:18:59 | gmann | melwitt: yeah that is ongoing fixes in devstack, and we have merged all the stable branch fix now | |
| 15:20:19 | dansmith | since monday I think | |
| 15:20:19 | melwitt | gmann: ah thanks, I missed that | |
| 15:20:41 | melwitt | I had a thing on stable (stein) fail zuul overnight | |
| 15:21:10 | gmann | yeah, | |
| 15:21:14 | melwitt | so I was wonderin | |
| 15:21:34 | gmann | but grenade multinode job is still issue, fixing in ussuri by backporting running zuulv3 native - https://review.opendev.org/#/c/735948/1 | |
| 15:22:51 | gmann | train and stein might need to make it n-v till we backport the grenade zuulv3 till stein. that is we are discussing in qa channel | |
| 15:25:39 | melwitt | ok I'm seeing neutron-grenade-multinode and nova-grenade-live-migration and grenade-py3 failing on stein | |
| 15:28:50 | melwitt | and then nova-live-migration is failing with "/opt/stack/new/devstack/lib/infra: line 32: virtualenv: command not found" on stein https://zuul.opendev.org/t/openstack/build/509288c2a75a49609c2f7f4b2e158a60/log/logs/devstack-subnodes-early.txt#2948 | |
| 15:33:01 | gmann | melwitt: yeah, that need devstack-gate fix which is not preferred so i am trying to backport the grenade zuulv3 till train. | |
| 15:45:09 | melwitt | gmann: ok, thanks. devstack-gate is branchless I assume? | |
| 15:45:27 | gmann | melwitt: yeah. | |
| 15:49:25 | melwitt | gmann: thanks. have you heard of the other failure before "/opt/stack/new/devstack/lib/infra: line 32: virtualenv: command not found"? for nova-live-migration | |
| 15:54:02 | gibi | melwitt: could this be due to http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015425.html ? | |
| 15:55:13 | melwitt | gibi: oh maybe, thanks. I missed that, was looking for "gate" in ML | |
| 15:55:19 | gmann | melwitt: that is same issue grenade multinode job facing | |
| 15:55:29 | gmann | and need fix in devstack-gate | |
| 15:56:04 | melwitt | oh, the error message was different so I didn't realize it was related | |
| 16:11:09 | TheJulia | dansmith: the goal is more, have the nova-compute driver ask ironic to hold the node, set appropriate values to represent the user has possesion of it and allow the user to do the needful. | |
| 16:11:41 | TheJulia | sean-k-mooney: the conundrum is the users expect to use nova, and currently get resources via nova today, and what they are doing is basically going and making internal invasive chagnes inside the deployed instance to re-install the instance using a kickstart | |
| 16:11:58 | dansmith | TheJulia: yeah that's the same as what I said, just with nova-compute being slightly involved in the enablement of the out-of-band activities | |
| 16:12:26 | TheJulia | Well, it may be logical to enable setting the "leasee" field in ironic's api anyway | |
| 16:12:39 | TheJulia | which would allow operators to optionally grant direct api rights | |
| 16:13:05 | TheJulia | I guess there could always be a trait that populates itself that just signals "don't actually deploy this machine, just allocate it" | |
| 16:13:15 | dansmith | that's akin to nova setting up a libvirt VM and then tolerating the user or admin mucking with the underlying VM XML | |
| 16:13:18 | TheJulia | and we could do that under the hood in ironic fairly easily. | |
| 16:13:21 | dansmith | which today we declare as "not okay" | |
| 16:15:12 | TheJulia | Well, we should still eventually store who asked for the thing and if we have community consensus behind a "delayed deployment" trigger, then that is totally up to the operators and their running policy to decide if they want to permit or not to meet their use cases | |
| 16:17:23 | sean-k-mooney | dansmith: for what its worth we had someone asking about pxe booting nova instance a few weeks ago | |
| 16:18:01 | sean-k-mooney | they were confused by the fact that if you enable the qemu boot menu via the image porperty you can select pxeboot as an option in the boot menu via the console | |
| 16:18:24 | sean-k-mooney | but could not figure out how to configure it which makes sense since we dont support you pxe booting the vms booted by nova | |
| 16:18:27 | dansmith | sean-k-mooney: you mean asking again | |
| 16:19:04 | sean-k-mooney | it came up on irc like a month ago i assuem it has come up before | |
| 16:19:09 | dansmith | many times | |
| 16:19:44 | dansmith | I dunno if there would be any ironic parallels here, but changing the image from inside the guest means that anything we do after that based on the image metadata will be wrong | |
| 16:20:06 | sean-k-mooney | i basically just said thats not supported sorry, it just a leaky abstraction | |
| 16:20:18 | dansmith | like attaching a device and using the correct device model, or migrating somewhere that looks at image metadata for OS properties, etc | |
| 16:20:26 | sean-k-mooney | dansmith: well so i have heard of people upgradeing form like 16.04 to 18.04 | |
| 16:20:37 | sean-k-mooney | but then you get into the pets vs cattle debate | |
| 16:20:55 | dansmith | which is wholly uncloudy, but is a lot less bad than going from 16.04 to Windows 10 | |
| 16:20:59 | dansmith | yup | |
| 16:21:46 | sean-k-mooney | ya updating the same disto is not really a problem different openrating systemsn entierely like linux to windows or esxi is a tolltal differnt matter | |
| 16:25:01 | sean-k-mooney | TheJulia: honestly it sound like they should update there kickstart playbook to delegate the os imagiang and provisioning to openstack/ironic and then just deal with boot straping the deployed hosts | |
| 16:26:13 | TheJulia | the case that is coming up at CERN is apparently they are deploying nodes with configurations that are impossible to articulate in images, so they are making the necessary major changes to get to their required end state after the fact | |
| 16:27:04 | sean-k-mooney | TheJulia: sure that makes sense but i dont know wny they would not just orcestrat the software payload via ansible and have a seperate role to deploy the orginal server | |
| 16:27:19 | sean-k-mooney | TheJulia: kind of like kaobe | |
| 16:27:27 | TheJulia | Really, ideally they should jsut be talking with ironic in some of these cases, but I guess they are trying to force their users into cloudy mindsets/patterns/behaviors | |
| 16:28:22 | dansmith | yup | |
| 16:28:33 | TheJulia | It is a mixed vm/baremetal environment, fwiw | |
| 16:29:14 | dansmith | what are the "node configurations that are impossible to articulate in images" ? | |
| 16:29:17 | TheJulia | And high level consistency seems to be the key they need, it is almost like power users just need that extra next step | |
| 16:29:22 | sean-k-mooney | ya that is not really an issue johnthetubaguy could proably tell you more about how they orcestrate the deployment of the Square Kilometer Array (SKA) | |
| 16:29:34 | TheJulia | multi-disk/raid/lvm/etc | |
| 16:29:51 | TheJulia | dansmith: this came out of the whole disk partitioning discussion getting resurrected | |
| 16:30:18 | dansmith | ack, well, addressing that makes more sense to me than just saying "short-circuit the bottom half of nova because it's easier" | |
| 16:30:54 | TheJulia | except, then get into the whole pets/cattle argument again and everyone tableflips and walks away | |
| 16:31:29 | sean-k-mooney | cern have a blog post on that but it seam sto be down the stack hpc one is up however https://www.stackhpc.com/software-raid-in-ironic.html | |
| 16:31:38 | dansmith | that's not a convincing argument to do the short-circuit :) | |
| 16:32:54 | TheJulia | dansmith: it is a solid reason people have downstream patches and are why they are not upgrading though, because their business processes need something along those lines. Anyway, if we enable them to short circuit in ironci, I think that is fine. Nova will believe it is happy. they can do whatever special config assertion as long as we populate leasee which we should be doing anyway. | |
| 16:33:53 | sean-k-mooney | TheJulia: there are ways to do per instance raid config today howere right | |
| 16:34:01 | TheJulia | which _should_ make the world a happier place. Dogs and cats could get married and all | |
| 16:34:51 | dansmith | whatever, it's just avoiding an abstraction because it's hard | |
| 16:34:52 | TheJulia | sean-k-mooney: it is not just raid sadly, the convergence seemed to be that there are teams with whole kickstart configs and to try and articulate that with traits will just make flavor/trait explosion even worse. | |
| 16:35:03 | dansmith | even though our only purpose in life is to be an abstraction | |
| 16:37:01 | TheJulia | in a perfect end state, I'd prefer them to get everything in glance and then tie it all together on an image basis, but smaller building blocks | |
| 16:37:28 | sean-k-mooney | TheJulia: well im not sure i agree that this should all live in glance or in the flavor | |
| 16:37:36 | sean-k-mooney | some of it should be user-data | |
| 16:38:00 | dansmith | it's not a smaller building block, it's a hammer to break the existing blocks | |
| 16:38:01 | dansmith | it doesn't build us toward a solution, it's an exit plan | |
| 16:38:43 | dansmith | instead of ever building new blocks, it's a forever solution of "get it to 40% complete and then do the rest custom out of band" | |
| 16:39:46 | sean-k-mooney | TheJulia: it sound like a way to pass info on a per instance basis to the ironic python agent would partly solve there issue. e.g. if the agent had a defien api of actions such as bios and raid config that could be pass on a per instance basis it would help with the image/flavor explosion issue | |
| 16:39:58 | TheJulia | woudl it help to have a larger discussion regarding this again, because where this always stalled out was the "how do we end up having something that nova will agree to" in the past in terms of data. and I think the last agreement of a possible solution involved templates stored in glance associated via traits or something which I think made it complex and cumbersome to interact with | |
| 16:40:19 | sean-k-mooney | the problem is that that for that to make sense it has to be somewhat abstract as we will not know the precise host ahead of time by design | |
| 16:40:55 | dansmith | we did have a long conversation about this in dublin, | |
| 16:41:03 | TheJulia | sean-k-mooney: we're actually have it, just turns out people want to do even more complex things their own way | |
| 16:41:10 | dansmith | which involved nova just knowing about a hardware template that gets passed to ironic to define these things | |
| 16:41:27 | dansmith | there was no tableflipping | |
| 16:41:54 | sean-k-mooney | dansmith: to do that we would want to do it in a way that did not invovle specifying the template in the flavor or image right | |
| 16:41:59 | TheJulia | maybe I was thinking of the time before | |
| 16:42:30 | sean-k-mooney | such as a swift url or glance artifcact uuid in server metadata or something else instance based | |
| 16:42:31 | dansmith | sean-k-mooney: I dunno, in the image seems fine to me | |
| 16:42:44 | sean-k-mooney | dansmith: in the image would work only to a point | |
| 16:42:55 | openstackgerrit | Alexandre Arents proposed openstack/nova stable/stein: Add admin doc information about image cache resource accounting https://review.opendev.org/732542 | |
| 16:43:07 | sean-k-mooney | if you assume the raid config is an atribute of that image then sure | |
| 16:43:31 | sean-k-mooney | but if the workload in that image has different performace configurtion it can be deployed in for different slas then you need multiple images | |
| 16:43:58 | dansmith | sean-k-mooney: how the user gets to select it (if at all) is really a smaller concern than all the components supporting that thing regardless of how it's presented | |
| 16:44:19 | sean-k-mooney | true | |
| 16:44:57 | TheJulia | I suspect your thinking that fitting the needs is a conundrum, but in these sorts of cases, the user knows exactly what the hardware is because they have a special flavor to use to match it | |
| 16:45:04 | dansmith | I tend to think that flavors and images are our templating mechanism and optimizing for a one-off deployment of one thing with weird config is not the direction we should go | |
| 16:45:29 | dansmith | TheJulia: right, which is why I think that in the flavor is reasonable in a lot of cases | |
| 16:46:00 | dansmith | they're already selecting an ironic machine class via flavor anyway | |
| 16:46:27 | TheJulia | From the flip side of that, another operator on the call we had today basically stressed that if they ahd to do that their flavors would explode even more and I think they already have tens of thousands | |
| 16:47:14 | sean-k-mooney | other then the fact openstack flavor list will be ... difficut to use did they have a partical problem with that | |
| 16:47:19 | TheJulia | (approximately 350,000 physical machines in their fleet is what they say) | |
| 16:47:33 | dansmith | I suspect that means they're doing it wrong, but regardless, if you literally have thousands of bespoke configurations, you're going to end up with thousands of incompatible objects somewhere in the system | |
| 16:47:53 | sean-k-mooney | TheJulia: but presumable that is grouped into a manageable set of SKUs | |
| 16:48:26 | dansmith | sean-k-mooney: that's why I said I expect they're doing it wrong.. they obviously have less than 350k unique configurations of machines, they probably just want to think of each machine as a snowflake | |
| 16:48:30 | sean-k-mooney | TheJulia: do you know how many unique ironic resouce classes they have | |
| 16:49:03 | TheJulia | sean-k-mooney: I'll ask, I don't know off the top of my head | |