Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-09-17
16:45:41 efried zzzeek: thanks for the pointer!
16:51:37 sean-k-mooney efried: i just commented on the mail but could the mysql issues be related too the the rabbit mq issue we have in the api
16:52:01 sean-k-mooney e.g. could it be cause by eventlet and running multiple thread in the wsgi server
16:52:16 sean-k-mooney im not sure if this is happeng in the gate or not
16:52:21 sean-k-mooney if its in the gate then no
16:52:23 efried sean-k-mooney: I have no. freaking. clue. If you can solve this one, I will buy you cookies forever.
16:52:42 efried it's happening in the gate, yeah.
16:52:54 sean-k-mooney ok in the gate we hard code one thread i think
16:52:57 sean-k-mooney but ill check
16:52:59 efried wait
16:53:04 efried we're talking about different issues, sorry.
16:53:16 efried disregard cookie promise
16:53:25 sean-k-mooney is was talking about RE: [oslo][nova] Nova causes MySQL timeouts
16:53:38 efried cookie promise applies to http://bugs.launchpad.net/bugs/1823251
16:53:39 openstack Launchpad bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed]
16:53:41 sean-k-mooney which i assme is what you and zzzeek were talking about?
16:53:51 efried no, zzzeek and I were talking about ----^
16:53:51 sean-k-mooney ah ok
17:16:11 efried sean-k-mooney: oh, it looks like it may actually be the same thing zzzeek and I were talking about.
17:16:18 efried which kind of forked off of the test_walk_versions thing.
17:17:02 sean-k-mooney the mysql timeouts
17:17:28 sean-k-mooney so i was wondering if it could be related to eventlet in genreal
17:18:28 sean-k-mooney connection pooling might not play nicely with evnetly monkey patching when don in a wsgi envionment
17:18:49 sean-k-mooney or rather a mod_wsgi or uwsgi env
17:19:36 sean-k-mooney that said if it was an eventlet issue i dont think it would affect the test_walk_versions stuff
17:21:09 efried sean-k-mooney: it may be entirely unrelated to the gate timeout in those tests. But I'm seeing it in the mysqld logs for those tests (whether they pass or fail). So, can't hurt to get rid of them, nah?
17:23:56 sean-k-mooney well i thik fixing mysql issue in general is a good thing
17:24:12 sean-k-mooney just not sure as you said fi they are releated or not
17:25:29 openstackgerrit François Palin proposed openstack/nova master: Add retry when attachment_delete or terminate_connection fails https://review.opendev.org/669674
17:51:23 openstackgerrit sean mooney proposed openstack/nova master: make config drives sticky bug 1835822 https://review.opendev.org/669738
17:51:24 openstack bug 1835822 in OpenStack Compute (nova) "vms loose acess to config drive with CONF.force_config_drive=True after hard reboot" [Medium,In progress] https://launchpad.net/bugs/1835822 - Assigned to sean mooney (sean-k-mooney)
17:52:39 mriedem_afk sean-k-mooney: i tagged the bug so we won't forget about it https://bugs.launchpad.net/nova/+bugs?field.tag=train-rc-potential
17:52:54 donnyd I fixed my mysql issues by removing the ulimits in systemd for the service
17:52:56 sean-k-mooney mriedem_afk: i added it to the etherpad
17:53:12 donnyd probably a better answer out there... but i haven't ran into it again
17:53:17 sean-k-mooney mriedem_afk: i think i have done the things you asked for
17:54:46 sean-k-mooney ah i see you are using launchpad to track the rc potentail bugs ya that is better
17:55:12 sean-k-mooney im going to grab food but ill asress your comments in the devname patch when i get back
18:05:13 mriedem sean-k-mooney: https://review.opendev.org/#/c/669738/5
18:37:47 efried donnyd: Were they the same mysql issues we've been discussing?
18:38:25 efried I sort of doubt it would be a problem to remove ulimits on CI nodes?
18:47:06 donnyd yea it was pretty close to the same issue
18:47:34 donnyd my load is different... but I just didn't want to hear any guff from mariadb
18:48:22 openstackgerrit Matt Riedemann proposed openstack/nova stable/rocky: lxc: make use of filter python3 compatible https://review.opendev.org/676498
18:48:31 donnyd woot woot LXC
18:56:16 sean-k-mooney donnyd: you will like this then https://review.opendev.org/#/c/667976/
18:56:32 donnyd YESSSS sean-k-mooney
18:57:13 sean-k-mooney i think i said this becore but we need to fix the console and resizeing of the root file system bit it more or less works
18:57:14 donnyd I am unreasonably pumped for LXC support... I think it works great and can't wait
18:57:35 sean-k-mooney well we supported lxc years ago
18:57:42 sean-k-mooney its just has not been maintianed
18:57:51 sean-k-mooney but i would like to see it working again
18:58:08 sean-k-mooney *fully working again
18:58:45 donnyd I know... and it worked great... LXC can support a large number of workloads and its heavily underused by those who just don't know... I use it every day
18:59:22 sean-k-mooney my main issue with lxc was the lack of a good way to build images for it
18:59:37 donnyd i think distrobuilder is growing
18:59:40 sean-k-mooney i first start to use docker when it use the lxc backend
18:59:51 donnyd there a quite a few image types available in there now
18:59:54 sean-k-mooney and that was perfoect form me
19:19:05 openstackgerrit Matt Riedemann proposed openstack/nova stable/queens: Drop source node allocations if finish_resize fails https://review.opendev.org/682722
19:23:31 donnyd I see nothing wrong with LXC
19:25:07 donnyd and additionally there are solid examples of people using it for prod workloads...It also makes me happy to see some work going into lxc to make it work again... Many thanks
19:31:42 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Add functional recreate test for regression bug 1825537 https://review.opendev.org/682725
19:31:43 openstack bug 1825537 in OpenStack Compute (nova) queens "finish_resize failures incorrectly revert allocations" [Medium,In progress] https://launchpad.net/bugs/1825537 - Assigned to Matt Riedemann (mriedem)
19:46:49 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Changing scheduler sync event from INFO to DEBUG https://review.opendev.org/669127
19:49:39 openstackgerrit Matt Riedemann proposed openstack/nova master: Refactor pre-live-migration work out of _do_live_migration https://review.opendev.org/641453
20:20:31 openstackgerrit Matt Riedemann proposed openstack/nova master: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/682741
20:36:36 gyee if the given vm is using pci passthrough, is resizing to the same host allowed or disallowed?
20:52:56 artom mriedem, left a question in https://review.opendev.org/#/c/682741/1
20:53:15 artom Figured it's more polite to ask on IRC than -1 the thing (even if that's what I initially did)
21:02:11 openstackgerrit Akihiro Motoki proposed openstack/nova master: doc: Improve PDF document structure https://review.opendev.org/682746
21:06:53 mriedem replied
21:07:46 artom Cheers
21:13:44 mriedem gyee: resize on the same host with pci passthrough should work but there is no integration testing for it, if there are limitations sean-k-mooney might be aware of them
21:14:32 mriedem i could see the code not being so smart and realizing that if pci devices in the flavor have not changed and we're resizing on the same host that we don't need to re-claim them
21:15:24 gyee mriedem, if the host has one PCI card, I am guessing we can't resize and it has already claimed
21:15:54 gyee but I am not too sure, have had time to dive into the code yet, was hoping for a quick yes or no answer :-)
21:16:18 mriedem i don't have that code memorized so i can't say off the top of my head
21:17:08 mordred mriedem: I'm very disappointed that you don't have it memorized
21:17:17 mriedem i'm not
21:17:18 gyee I see this in the logs "nova-scheduler.log:2019-09-17 19:41:34.269 31312 DEBUG nova.scheduler.filters.pci_passthrough_filter [req-9165ec70-88b2-4ce1-804b-ee28d7c1992c c6d51320a59041588a26477c64ec0973 d35583e194d845b79aaa2a6de2bef94b - default default] (d14-58-d0-52-4c-28, d14-58-d0-52-4c-28.i1.cloud.suse.de) ram: 114519MB disk: 1468416MB io_ops: 0 instances: 2 doesn't have the required PCI devices (InstancePCIRequests(i
21:17:18 gyee nstance_uuid=<?>,requests=[InstancePCIRequest])) host_passes /usr/lib/python2.7/site-packages/nova/scheduler/filters/pci_passthrough_filter.py:54"
21:17:25 mriedem gyee: the claims code in the resource tracker would be here for resize https://github.com/openstack/nova/blob/0984ebb9f1c3cc3c8ff352dd2a615a58da79833c/nova/compute/resource_tracker.py#L280
21:17:52 gyee so I am guessing the device has already be claimed and therefore we can't share the same device, even on resizing
21:19:06 mriedem gyee: the PciPassthroughFilter might not be smart enough for resize to the same host yeah
21:19:20 mriedem in general the scheduler and its filters don't know about move operations
21:19:44 mriedem mordred: also, what a weird time for you to show up and heckle me :)
21:21:27 mriedem gyee: i wouldn't be surprised if you look at the starlingx nova repo that they have something for pci passthrough same host resize
21:22:00 mriedem cfriesen: ^
21:23:02 mordred mriedem: it's 11:22pm where I am right now - that seems like perfect heckling time
21:23:04 gyee mriedem, thanks for the info. Time to read code I guess. :-)
21:24:23 mriedem mordred: so you're bored and lonely eh
21:24:52 mordred mriedem: well, there's no aurora borealis tonight
21:26:28 cfriesen mriedem: gyee: I think we're now aligned with upstream nova now, and that doesn't ring any bells for me. Checking with someone.
21:28:12 mriedem i would think being able to resize a pci passthrough instance on a single node edge site would be important for starlingx
21:28:37 gyee cfriesen, thanks. Our instance is running stable/rocky right now, if that makes any difference.
21:41:50 cfriesen gyee: mriedem: just checked and can't find anything like that. Since Stein at least Starlingx has been aligned with upstream nova. Sounds like a useful change though.
21:45:23 gyee cfriesen, thanks for the info! Yeah, I agree, resizing to the same host is a useful feature
22:27:07 cfriesen gyee: I did some more digging, turns out that we did do some special-casing for sriov and resize/cold migration for Pike, because cold migration with PCI passthrough was broken. For Stein, I think the only customization is numa-aware live migration.
22:45:12 openstackgerrit Merged openstack/nova stable/stein: Fix 'has_calls' method calls in unit tests https://review.opendev.org/677329

Earlier   Later