| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-05 | |||
| 15:46:17 | fried_rice | sean-k-mooney This in Queens?? | |
| 15:46:18 | mriedem | johnthetubaguy: orly | |
| 15:46:24 | mriedem | Jan 04 20:06:18.133057 ubuntu-xenial-rax-ord-0001690099 nova-scheduler[9487]: WARNING nova.scheduler.filters.compute_filter [None req-370806fa-5640-485d-adc2-21808de9ccb4 tempest-MigrationsAdminTest-1101936831 tempest-MigrationsAdminTest-1101936831] (ubuntu-xenial-rax-ord-0001690099, ubuntu-xenial-rax-ord-0001690099) ram: 6446MB disk: 58368MB io_ops: 15 instances: 15 has not been heard from in a while | |
| 15:46:24 | mriedem | yeah because i'm seeing this in the scheduler logs | |
| 15:46:28 | mriedem | but no errors in the n-cpu logs | |
| 15:46:31 | hrw | melwitt: o yes ;D | |
| 15:46:46 | johnthetubaguy | mriedem: if your cache interval is larger than than how long you thing it is dead, it starts to ignore some of your nodes randomly | |
| 15:46:46 | hrw | melwitt: can you take a look at https://review.openstack.org/#/c/530965/ too? | |
| 15:46:54 | hrw | melwitt: I am open for any questions | |
| 15:46:54 | mriedem | ah | |
| 15:46:57 | mnaser | mriedem: i think i found out how i got that weird state. doing a host-evacuate-live sends a live-migration request for all instances, as nova processes them one by one, the states of all vms in the compute host are in "migrating".. but then its possible that a delete request comes in while the live migration is happening. | |
| 15:47:03 | sean-k-mooney | fried_rice: not upstream, libvirt and qemu dont even have support for vhost-vfio yet. way to early to propose upstrem in openstack | |
| 15:47:16 | melwitt | hrw: I'll add it to my list | |
| 15:47:21 | hrw | melwitt: thanks | |
| 15:47:42 | hrw | melwitt: with those two merged getting openstack on aarch64 is much easier ;D | |
| 15:48:10 | mnaser | in the rpc code, there doesn't seem to be any handling if the VM is in migrating state | |
| 15:48:12 | sean-k-mooney | fried_rice: i would like to have a poc working ptg/early rocky timeframe but i doubt the dependcies will align for it to land even in rocky | |
| 15:48:16 | mnaser | for deletes | |
| 15:48:31 | hrw | melwitt: uefi change is most important as with it in nova we cut 90% of users questions ;d | |
| 15:48:58 | kashyap | mriedem: Hey was AFK | |
| 15:49:00 | fried_rice | sean-k-mooney Okay, gotcha. You going to be around for the sched meeting on Monday? This is exactly the kind of rationale cdent and Radoslav were bringing up in https://review.openstack.org/#/c/526539/ (to pose the question, "Why are we artificially restricting in-tree virt drivers from talking directly to placement?" | |
| 15:49:03 | kashyap | Reading the scroll | |
| 15:49:15 | fried_rice | ) | |
| 15:49:41 | mriedem | johnthetubaguy: i don't see a specific config option for how long the cachingscheduler caches it's host state information | |
| 15:49:50 | mriedem | is that somewhere outside the actual CachingScheduler class? | |
| 15:49:54 | sean-k-mooney | fried_rice: am what time is it?( checking meeting page) | |
| 15:50:05 | johnthetubaguy | mriedem: ah, I think its in scheduler manager, sadly | |
| 15:50:20 | johnthetubaguy | just digging | |
| 15:50:34 | mriedem | self.driver.run_periodic_tasks(context) | |
| 15:50:34 | mriedem | def _run_periodic_tasks(self, context): | |
| 15:50:34 | mriedem | run_immediately=True) | |
| 15:50:34 | mriedem | @periodic_task.periodic_task(spacing=CONF.scheduler.periodic_task_interval, | |
| 15:50:35 | mriedem | ? | |
| 15:50:50 | johnthetubaguy | mriedem, yeah https://github.com/openstack/nova/blob/d19e1252d5f2f8e8b54a276003f842ee00641026/nova/scheduler/manager.py#L79 | |
| 15:50:51 | mriedem | yup that's it | |
| 15:51:08 | sean-k-mooney | fried_rice: 14:00 UTC i should be able to attend. | |
| 15:51:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: introduce framework for RFB authentication https://review.openstack.org/345397 | |
| 15:51:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: Provide an RFB security proxy implementation https://review.openstack.org/345399 | |
| 15:51:14 | openstackgerrit | Stephen Finucane proposed openstack/nova master: console: introduce the VeNCrypt RFB authentication scheme https://review.openstack.org/345398 | |
| 15:51:15 | openstackgerrit | Stephen Finucane proposed openstack/nova master: doc: Document TLS security setup for noVNC proxy https://review.openstack.org/500544 | |
| 15:51:16 | fried_rice | sean-k-mooney Cool. | |
| 15:51:23 | leakypipes | cdent, fried_rice: see my comment on https://review.openstack.org/#/c/526539/ | |
| 15:51:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: [libvirt] Allow multiple volume attachments https://review.openstack.org/267587 | |
| 15:51:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: [api] Allow multi-attach in compute api https://review.openstack.org/271047 | |
| 15:51:44 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass multiattach flag to reserve_block_device_name https://review.openstack.org/531244 | |
| 15:51:54 | johnthetubaguy | mriedem: trying to find the docks I wrote about this, failing :( | |
| 15:53:10 | stephenfin | mriedem: Could you hit this again at some point today? Got an open question on it for you https://review.openstack.org/#/c/526329/ | |
| 15:53:17 | mriedem | the config option help for that actually points out the service_down_time option too | |
| 15:53:19 | cdent | leakypipes: thanks reading. I think I tend to use "arbitrary" incorrectly or at least with different connotations that most peole do. I tend to mean it as "something we chose" (as opposed to "a thing we must do") | |
| 15:53:22 | mriedem | so i'm pleasantly surprised | |
| 15:53:29 | johnthetubaguy | mriedem: https://github.com/openstack/nova/blob/d19e1252d5f2f8e8b54a276003f842ee00641026/nova/conf/scheduler.py#L78 | |
| 15:53:39 | johnthetubaguy | mriedem: yeah, I knew I put that somewhere! | |
| 15:53:43 | mriedem | johnthetubaguy: yup, was just reading that | |
| 15:53:44 | mriedem | nice | |
| 15:53:49 | stephenfin | oh | |
| 15:54:19 | leakypipes | cdent: sure, understood. | |
| 15:54:30 | sean-k-mooney | fried_rice: im hoping i wont actully need to modify the virt driver as part of the poc beyond the xml generation. on of the advantages of mdevs is you create them using arbitray uuids(neutron port uuid in my case) dynamically so i may be able to skip the resouce tracker entrily as i basically just need a count of howmany can be created per host which the placement inventories will already have. | |
| 15:55:35 | johnthetubaguy | mriedem: there is a type there s/scheduler service as down/compute service as down/ | |
| 15:55:36 | fried_rice | leakypipes cdent Good deal. I can see both sides. I don't mind the idea of having intra-nova placement work funneled through this spot for the reasons stated (thread safety, code commonality) as long as we're not doing it with the expectation that e.g. OOT drivers have to conform. | |
| 15:55:59 | fried_rice | sean-k-mooney So where are you going to get that request info from? Just directly from the flavor? | |
| 15:56:29 | mriedem | finucannot: replied | |
| 15:56:48 | mriedem | finucannot: we could backport that to stable/pike upstream, but no further given ocata g-r requires glanceclient>=2.5 | |
| 15:56:57 | mriedem | johnthetubaguy: yup i noticed that too | |
| 15:57:07 | mriedem | johnthetubaguy: i'm going to remove the deprecated names for that option, so i'll fix the typo at the same time | |
| 15:57:11 | leakypipes | fried_rice: well, it's Python so of course OOT drivers can always do whatever they want with things ;) but that said, yes, I'd prefer to have the virt drivers just update the ProviderTree object and rely on something else (the reportclient called from the resource tracker most likely) do the work of making placement REST API calls. | |
| 15:57:21 | finucannot | mriedem: Ta | |
| 15:57:26 | johnthetubaguy | mriedem: sweet, thank you | |
| 15:57:27 | sean-k-mooney | fried_rice: of course when i try to do that perhaps ill discovery why mriedem thinks ill what to stabs ye lol. am the requst for the port is encode by the new vnic type + neutron port bining profile info for mdev_type if needed. | |
| 15:59:29 | fried_rice | sean-k-mooney IMO what you're doing here is closer to what should be happening long term. It would be nice if the scheduler and resource tracker could accept requests for resources, allocate them via placement, and pass those allocations through to virt without introspecting or special-casing them at all. | |
| 15:59:54 | mriedem | johnthetubaguy: actually, a bit confused | |
| 16:00:18 | fried_rice | sean-k-mooney I.e. compute code outside of virt doesn't need to care if a resource is a CPU, GPU, mdev (whatever that is), or peanut. | |
| 16:00:18 | mriedem | johnthetubaguy: it says, "This is because the scheduler driver is responsible for sending a heartbeat and it will only do that as often as this option allows." | |
| 16:00:29 | mriedem | johnthetubaguy: i don't see how the compute service down time and the scheduler periodic task interval are linke | |
| 16:00:30 | mriedem | *linked | |
| 16:00:53 | mriedem | sean-k-mooney: you will get stabby because of the constant spelling corrections | |
| 16:01:13 | mriedem | like, "spell knife in the kidneys!" | |
| 16:01:58 | fried_rice | Oh, did I do that? Not consciously. | |
| 16:02:05 | sean-k-mooney | fried_rice: yes i had hopped to work on this as part of the nova neutron negociation stuff in queens and the multiple port binding before we change from upstream first to more poc/internal focoused. | |
| 16:03:16 | sean-k-mooney | mriedem: haha i would have taught i would be the other way round. jay has very kindly help me with this area over the past few years and im very greatful for that. | |
| 16:03:35 | openstackgerrit | Ed Leafe proposed openstack/nova master: Change compute RPC to use alternates for resize https://review.openstack.org/526436 | |
| 16:03:35 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add functional test for resize reschedules https://review.openstack.org/531022 | |
| 16:03:36 | openstackgerrit | Ed Leafe proposed openstack/nova master: Add regression test for resizing failing during retries https://review.openstack.org/531405 | |
| 16:03:54 | figleaf | mriedem: ^^ added regression test and fix | |
| 16:04:09 | mriedem | cool | |
| 16:04:12 | leakypipes | sean-k-mooney: you would have *thought*. | |
| 16:04:16 | leakypipes | sean-k-mooney: you're welcome. | |
| 16:04:28 | sean-k-mooney | leakypipes: :) | |
| 16:04:39 | leakypipes | sean-k-mooney: also... it's *grateful", not greatful. you're welcome, too. | |
| 16:04:49 | leakypipes | sean-k-mooney: :P | |
| 16:05:43 | kashyap | mriedem: Hope you recorded your observation about raw & the shareable thing in the multi-attach review | |
| 16:06:00 | openstackgerrit | melanie witt proposed openstack/nova stable/pike: libvirt: Re-initialise volumes, encryptors, and vifs on hard reboot https://review.openstack.org/531407 | |
| 16:06:01 | kashyap | Currently, I'm in split-brains mode, and trying to finishing something before the 'cache is flushed | |
| 16:06:43 | mriedem | kashyap: i did, and it was just something i should have realized wouldn't apply here earlier on | |
| 16:06:47 | mriedem | since imagebackend != volumes | |
| 16:08:03 | mriedem | johnthetubaguy: oh i think i see, | |
| 16:08:20 | mriedem | the ComputeFilter is running every 60 seconds due to the caching scheduler's periodic task interval, | |
| 16:08:20 | figleaf | fried_rice: sorry, missed your question earlier - network issues. | |
| 16:08:31 | mriedem | which means it's potentially using a stale version of the compute service reference, | |
| 16:08:33 | figleaf | fried_rice: mriedem answered what I would have said, so no worries | |
| 16:08:36 | johnthetubaguy | mriedem: sorry, yeah, that's it | |