| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-09-19 | |||
| 15:21:52 | mriedem | we should probably identify that a problem exists before discussing designs on how to fix it | |
| 15:21:57 | bauzas | probably | |
| 15:22:02 | bauzas | and I'm unclear | |
| 15:22:25 | bauzas | I'll just leave a comment in my patch and I move on | |
| 15:28:27 | dansmith | mnaser: how much of your two minutes is writing and reading that dumpfile on disk? would you prefer we just pipe it between the export and import (optionally?) and maybe tee it out for forensics? might be faster | |
| 15:30:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option https://review.openstack.org/603831 | |
| 15:31:02 | mriedem | gmann: ^ | |
| 15:34:51 | mnaser | dansmith: like 3-5 seconds for the dump. The write took the longest | |
| 15:35:48 | dansmith | mnaser: 3-5 seconds overhead for writing the file? or 3-5 seconds to do the dump and the rest of the minute to write it out? | |
| 15:36:00 | dansmith | on my tiny database it's immeasurable of course | |
| 15:36:14 | dansmith | I get a constant 3.6s to do the dump and import regardless | |
| 15:36:35 | dansmith | mnaser: I have a diff that does it pipely if you want to try it | |
| 15:36:48 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove deprecated hide_server_address_states option https://review.openstack.org/603831 | |
| 15:37:32 | dansmith | mnaser: https://termbin.co/FXJg | |
| 15:38:05 | dansmith | I also put a "tee $tmpfile |" in the middle of that pipe (hence the comment) but removed it for your test | |
| 15:56:09 | openstackgerrit | Ben Nemec proposed openstack/nova master: WIP: Migrate upgrade checks to oslo.upgradecheck https://review.openstack.org/603499 | |
| 16:00:26 | dansmith | tssurya: mriedem shall we meet about cells? we just talked about things last week and melwitt is not around this week | |
| 16:01:44 | mnaser | dansmith: sorry i was in a meeting, the dump was really quick but the push was much slower because of (what i assume) replication | |
| 16:01:58 | dansmith | ah the load, I see | |
| 16:02:08 | dansmith | yeah I'm sure that's the heavy part | |
| 16:02:16 | dansmith | it's heavy on my toy devstack even | |
| 16:03:05 | dansmith | in that case, I'd just say we should leave separate the dump/load like it is now so that one can complete without the other, leaving a file you can just import if you want it | |
| 16:05:15 | jaypipes | stephenfin: so, unless I'm mis-reading your comments on the cpu-resource-tracking spec, you'd actually favor an "opt-in" approach to hyperthread usage. Is that correct? i.e. the virt driver defaults to *not* counting hyperthreads as CPU resources for guests unless some knob is turned on? | |
| 16:06:06 | stephenfin | jaypipes: For instances with dedicated CPUs, yes. However, we need to avoid breaking users | |
| 16:06:13 | stephenfin | So that knob would have to default to on | |
| 16:06:58 | stephenfin | The reason I want that is because a hyperthread != a core and I think the decision to model it as one was a mistake on day one | |
| 16:06:59 | jaypipes | stephenfin: ack. no disagreement from me on that | |
| 16:07:12 | jaypipes | stephenfin: agreed. | |
| 16:10:21 | openstackgerrit | sean mooney proposed openstack/nova-specs master: [WIP] generic device discovery policy https://review.openstack.org/603805 | |
| 16:20:37 | openstackgerrit | Merged openstack/nova stable/queens: Fix DB archiver AttributeError due to wrong table name attribute used https://review.openstack.org/599882 | |
| 16:20:44 | openstackgerrit | Merged openstack/nova stable/queens: VMware: fix TypeError while get console log https://review.openstack.org/591365 | |
| 16:20:51 | openstackgerrit | Merged openstack/nova stable/queens: Move conductor wait_until_ready() delay before manager init https://review.openstack.org/599200 | |
| 16:28:59 | mriedem | dansmith: i'm good to skip; plan on starting my poc for cross-cell resize today | |
| 16:29:11 | dansmith | oh boy | |
| 16:30:33 | openstackgerrit | Merged openstack/nova stable/ocata: Return 400 when compute host is not found https://review.openstack.org/590649 | |
| 16:31:38 | stephenfin | jaypipes: If you don't have one already, a placement cheatsheet (in PDF and therefore printable) would be super useful | |
| 16:31:58 | jaypipes | stephenfin: also, I agree with you about getting rid of emulator_threads_policy eventually, but is it 100% critical that I update the cpu-resource-tracking spec with information about that? | |
| 16:32:24 | jaypipes | stephenfin: you mean cheatsheet about the REST API? | |
| 16:32:29 | jaypipes | stephenfin: or something else? | |
| 16:32:41 | jaypipes | stephenfin: or you mean a glossary of placement terms? | |
| 16:33:26 | stephenfin | jaypipes: No, it can be be handled separately. However, I imagine you do need to take cpu overhead, which is incremented for each dedicated emulator thread core, into account | |
| 16:33:31 | openstackgerrit | Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds/reboots https://review.openstack.org/603844 | |
| 16:33:55 | stephenfin | jaypipes: Both. Glossary of terms followed by examples of either the REST API or using osc-placement | |
| 16:34:17 | jaypipes | stephenfin: ack | |
| 16:34:21 | stephenfin | jaypipes: Just an idea, obviously. I might even start drafting something myself some point this week | |
| 16:34:44 | jaypipes | stephenfin: cool. let me know when you get to the "what is a consumer?" part... | |
| 16:36:43 | stephenfin | jaypipes: FYI, the reason I brought up the whole "let's kill 'isolate' idea" was that that approach seemed easier than handling the CPU overhead thing *and* it reduced complexity for the user in the process, which feels like one of the goals | |
| 16:37:26 | stephenfin | But again, not a blocker :) | |
| 16:39:07 | jaypipes | stephenfin: yes, agreed. glad it's not a blocker though. | |
| 16:39:27 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: libvirt: mdevs returning parent and vendor PCI info https://review.openstack.org/562304 | |
| 16:40:21 | bauzas | jaypipes: stephenfin: can one of you reapprove https://review.openstack.org/562304 ? I just rebased it out of the existing series and I need it for the reshaper change | |
| 16:40:52 | bauzas | I'm surprised I lost my +2s on the rebase but meh | |
| 16:41:02 | bauzas | probably because I changed the commit id | |
| 16:41:15 | bauzas | oh yeah, I changed the message, hence why | |
| 16:41:30 | jaypipes | stephenfin: feel free to re-+W bauzas patch | |
| 16:41:44 | bauzas | jaypipes: thanks | |
| 16:42:15 | bauzas | I need the parenting relationship for reshaping the VGPU resources to the right PGPU RP | |
| 16:42:54 | stephenfin | bauzas: Done | |
| 16:43:15 | bauzas | stephenfin: ack, thanks | |
| 17:01:41 | openstackgerrit | Jay Pipes proposed openstack/nova-specs master: Standardize CPU resource tracking https://review.openstack.org/555081 | |
| 17:09:08 | tobias-urdin | ping on a possible upgrade path bug q->r https://bugs.launchpad.net/nova/+bug/1793353 | |
| 17:09:09 | openstack | Launchpad bug 1793353 in OpenStack Compute (nova) "broken upgrade path q->r requirement for oslo.db" [Undecided,New] | |
| 17:37:11 | openstackgerrit | David Rabel proposed openstack/nova master: Really use source image format as default for snapshot_image_format https://review.openstack.org/603855 | |
| 17:51:23 | jiteka | hello, could someone confirm that migrate feature is working with flavor using vCPU pinning and Numa in Mitaka ? | |
| 18:01:58 | openstackgerrit | Merged openstack/nova-specs master: Propose configurable maximum number of volumes to attach https://review.openstack.org/597306 | |
| 18:10:49 | mriedem | jiteka: cfriesen might know | |
| 18:11:09 | mriedem | there were some patches that mellanox and windriver worked on to make that work, but i can't remember which release in which those patches landed | |
| 18:11:28 | mriedem | jiteka: cold migrate i mean, not live migration | |
| 18:11:36 | mriedem | numa/pinned cpus does not work with live migration | |
| 18:11:53 | mriedem | https://specs.openstack.org/openstack/nova-specs/specs/rocky/approved/numa-aware-live-migration.html | |
| 18:28:40 | openstackgerrit | Merged openstack/nova stable/ocata: Add recreate test for RT.stats bug 1784705 https://review.openstack.org/588076 | |
| 18:28:40 | openstack | bug 1784705 in OpenStack Compute (nova) ocata "ResourceTracker.stats can leak across multiple ironic nodes" [High,In progress] https://launchpad.net/bugs/1784705 - Assigned to Matt Riedemann (mriedem) | |
| 18:31:27 | mriedem | dansmith: would be good to get this pike backport in https://review.openstack.org/#/c/599883/ | |
| 18:32:14 | dansmith | mriedem: hmm, did I do that? ISTR some reasoning there.. maybe sqla version? | |
| 18:32:59 | mriedem | it was originally your code yes, but table.name is standard | |
| 18:44:04 | mriedem | dansmith: does anything jump out at you for the source of the IndexError in this failure? http://logs.openstack.org/72/600372/1/gate/openstack-tox-py35/dcfd363/job-output.txt.gz | |
| 18:44:56 | mriedem | looks like something between oslo.db/sqla/pymysql/eventlet switches context and we timeout after some huge amount of time | |
| 18:45:07 | mriedem | nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [664.512994s] ... FAILED | |
| 18:45:11 | mriedem | nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_models_sync [39.644814s] ... ok | |
| 18:45:30 | mriedem | clearly we're not just missing some short timeout window | |
| 18:46:09 | dansmith | still waiting for the damn log to load | |
| 18:48:14 | dansmith | wow | |
| 18:48:54 | mriedem | yeah we have a lot of these failures in the gate across master, rocky and queens, | |
| 18:48:55 | mriedem | https://bugs.launchpad.net/cinder/+bug/1793364 | |
| 18:48:55 | openstack | Launchpad bug 1793364 in OpenStack Compute (nova) "mysql db opportunistic unit tests timing out intermittently in the gate (bad thread switch?)" [High,Confirmed] | |
| 18:49:05 | mriedem | nova and cinder b/c they are using the same test fixtures | |
| 18:49:55 | dansmith | hmm | |
| 18:50:30 | openstackgerrit | Eric Fried proposed openstack/nova master: Add contributor guide for upgrade status checks https://review.openstack.org/596902 | |
| 18:50:39 | mriedem | maybe "connection.scalar(select([1]))"? | |
| 18:50:42 | dansmith | does the index complaint surprise you? because it looks like a nonsense name | |
| 18:51:08 | mriedem | well i see this too | |
| 18:51:09 | mriedem | sqlalchemy.exc.ResourceClosedError: This result object does not return rows. It has been closed automatically | |
| 18:51:27 | dansmith | yeah | |
| 18:52:34 | mriedem | http://status.openstack.org/elastic-recheck/data/integrated_gate.html Overall Categorization Rate: 15.4% | |
| 18:52:47 | mriedem | meaning we (openstack) have a shit load of uncategorized failures killing stuff in the gate | |
| 18:52:55 | mriedem | which is why it's taking us days to merge code | |
| 18:57:32 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22%20%20connection.scalar(select(%5B1%5D))'%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 19:04:33 | cfriesen | jiteka: I think cold migration should work in mitaka. I think there are some gotchas where resource tracking isn't accurate until the next audit, and maybe there are some resource tracking issues if you revert a resize. | |
| 19:06:13 | tssurya | dansmith: ack, let's skip | |
| 19:06:16 | cfriesen | jiteka: a bit more digging shows that PCI devices were not cold migratable until Newton. | |